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ABSTRACT 


The Combined Arms Analysis Directorate of the Training and Doctrine Analysis Command 
(TRAC) uses the Analytic Hierarchy Process (AHP) to evaluate the contribution of modernization 
initiatives to U.S. Army capabilities. This thesis identifies several problems with using AHP. 
Most significantly, AHP can cause rank reversal of alternatives if a new alternative is considered, 
even if the new alternative has the same attribute levels as one of the previous alternatives. This 
thesis proposes several modifications that would improve results when AHP is used. It contains 
a different method of weight fitting that appears to provide alternative weights that are more 
accurate than the traditional AHP eigenvalue method. This thesis has two proposals for 
improving the nine point integer scale by which pairwise Comparisons are made. Most 
significantly, this thesis proposes a modification to AHP that will maintain a ratio scale and avoid 
rank reversals. This last improvement requires the decision maker to establish and maintain units 
of measurement. Additionally, the decision maker must make comparisons of attributes to 
establish a meaningful scale not sensitive to the abundance or lack of alternatives considered. 
If units are maintained and the decision maker is consistent in the pairwise Comparisons, there 


will be no rank reversals. 
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EXECUTIVE SUMMARY 

The Combined Arms Analysis Directorate of the Training and Doctrine Analysis Command 
(TRAC) uses the Analytic Hierarchy Process (AHP) to evaluate the level of future U.S. Army 
modernization that is achieved by management decision packages. 

A study of AHP shows it contains several flaws. First, and of greatest concern, AHP can 
allow rank reversals among alternatives if another alternative is considered, or if one is removed. 
This can occur even if the additional alternative has the same attribute levels as one of the original 
alternatives. 

We use a decision problem which evaluates tanks to illustrate AHP’s greatest flaw. Assume 
that a tank is measured by two attributes, survivability and firepower, and that the decision maker 


determined firepower is 1.25 times as important as survivability. To maintain that preference and 


normalize weights to sum to one, weights must be Ene) for survivability and firepower, 


respectively. 
Tank characteristics: 
(1) Survivability. On comparing three types of armor, reactive armor is preferred 1.5 
over applique, and applique is preferred 2 over rolled homogeneous armor. 
(2) Firepower. A tank with a 130mm main gun is preferred 1.4 over a tank with a 
120mm main gun. A 120mm main gun is preferred 1.5 over a 105mm main gun. A 105mm main 


gun is preferred 2.0 over a 90mm main gun. 


Viil 


Initially, we compare two tanks using AHP. Tank | has applique armor and a 120mm main 


gun. Tank 2 has reactive armor and a 105mm main gun. 


ATTRIBUTES RELATIVE 


VALUE 
SURVIVABILITY FIREPOWER 
(4/9) (5/9) 





Figure 1. Comparison of two tanks using AHP. 
The relative value says one would prefer Tank 1 to Tank 2. That is 
Tank 1 > Tank 2. 
In addition to the above two tanks, we now consider a third tank. Tank 3 has rolled 


homogeneous armor and a 130mm main gun. The result of this comparison is disturbing. 


ee es 


ATTRIBUTES RELATIVE 
TANK VALUE 
SURVIVABILITY FIREPOWER 
(4/9) (S/ 9) 


TANK | | 0.333 0.326 GW. 329 


TANK 2 0.217 0.343 
TANK 3 0.167 0.457 0.328 


Figure 2. Comparison of three tanks using AHP. 





Even though the decision maker did not change relative weights of firepower or survivability, or the 
characteristics of the first two tanks, AHP tells the decision maker his preferences have changed. 
That is 
Tank 2 > Tank 1 > Tank 3. 
Our modification, illustrated using the same tank problem described above, solves the problem 
of rank reversal. In our modification, the survivability attribute of Tank 1 is selected as the least 


preferred alternative in survivability. The firepower attribute of Tank 1 must remain 1.5 times the 


magnitude of the suvivability attribute. The firepower attribute of Tank 2 (105mm main gun) is 
determined to be equivalent to the survivability attribute of Tank 1 (applique armor). These 


restrictions are essential to maintaining a ratio scale. 


| ATTRIBUTES si pee 
TANK 
SURVIVABILITY FIREPOWER 
(4/9) (5/9) 


| TANK 1 | 1.2778 


Te 1.2222 


Figure 3. Modification used on original tank problem. 









The result 1s consistent with the result from using AHP in the orginal problem. That ts 


Tank 1 > Tank 2. 


Now, we include the third tank, Tank 3. 









| ATTRIBUTES BUTES RELATIVE 
i VALUE 
| SURVIVABILITY FIREPOWER 
(4/9) (5/9) 
TANK 1 lr. 2798 


TANK 2 hs 
TANK 3 1.3889 


Figure 4. Modification with additional alternative included, Tank 3. 
We maintain the notional “units” from the two tank problem. That is, Tank 1 survivability is the 
"unit" for survivability and the equivalency of Tank 1’s survivability is maintained with the value of 
the firepower of Tank 2. The result of this consistency in "units" is the preference 
Tank 3_> Tank 1 > Tank 2. 
There is no rank reversal. 
Also, AHP uses an integer nine point scale, with reciprocals possible, that severely limits the 


decision maker’s ability to evaluate possible alternatives and attributes in a decision problem. 


Finally, AHP uses the eigenvalue method of weight fitting with little justification. The weight 
fitting by the eigenvalue method was consistently (nine cases) less accurate than a simple least squares 
error (LSE) evaluation method. 

Despite the claim of the developer of AHP, Thomas L. Saaty, rank reversals are possible even 
if the new alternative is not within 10% of original alternatives in attributes. Since rank reversal can 
occur, AHP cannot maintain a ratio scale, a claim that it makes and that is desirable when output of 
AHP is used as input to optimization models. 

This thesis proposes modifications to AHP that address the above problems. First, it is clear 
that the restriction on scaling alternatives to integers with a maximum value of nine exacerbates the 
problems of inconsistency by a decision maker. There are two proposals in this area. First, the 
decision maker could identify the extreme attributes and extreme alternatives by attribute. Assign the 
lowest or least preferred a value of one, and the highest or most preferred a value of nine. In this 
way, all attributes and alternatives will be within the nine point scale. Alternatively, the decision 
maker could use an open ended scale. 

The second proposal for modification of AHP suggests an alternative weight fitting method. 
The method of least square errors (LSE) for weight fitting produces an exact solution to alternative 
weights if the decision maker is perfectly consistent, just as AHP’s eigenvalue method will. The LSE 
method appears to have an advantage in the likely case of decision maker inconsistency. 

The third modification proposal involves solving the problem of rank reversals while 
maintaining a ratio scale. The decision maker can assign the most preferred alternative in one 
attribute a value of one, and assign all other alternatives a value of less than one. The second option 
is to assign the Jeast preferred alternative in one attribute a value of one. If this standard is 
maintained and the decision maker is consistent in all pairwise comparisons, there can be no rank 


reversal. 
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I. BACKGROUND 


A. PROBLEM 

The Combined Arms Analysis Directorate of the Training and 
Doctrine Analysis Command (TRAC) at Fort Leavenworth, Kansas 
requested assistance in determining values for modernization 
initiatives. There is concern that the Analytic Hierarchy 
Process (AHP), the system currently used, might not be the 
best method to generate relative values of modernization 


imitiatives. 


B. SITUATION 

The United States Army Training and Doctrine Command 
(TRAC) at Fort Leavenworth, Kansas developed a new analytical 
decision Support system to help the Army develop a 
modernization investment program that reflects the values of 
its senior leadership. TRAC developed the Research 
Development and Acquisition Alternatives Analyzer (RDA3) to 
determine values for candidate modernization initiatives by 


mathematically optimizing allocation of resources. 


C. CURRENT SYSTEM 
1. Description of RDA3 
The RDA3 decision Support system involves two distinct 


phases. 


a. Hierarchal Assessment 

In the first phase, modernization initwvarive. 
(MDEPs) are determined through a hierarchal assessment. The 
overall goal (Future Army Modernization) is at the top of this 
hierarchy, and modernization initiatives can be several layers 
down this hierarchy through layers of supporting goals and 
objectives, sub-goals and sub-objectives. The goal is the top 
level of the hierarchy (Future Army Modernization). The first 
level of the hierarchy consists of 14 Army Mission Areas. 
Through a series of comparative judgement questions, the 
respondent or decision maker assesses the importance of each 
Mission Area to Army modernization. Similarly, sub-areas are 
assessed for their contribution to Army Mission Areas. This 


process continues until the bottom level of the hierarchy is 


reached. The bottom layer consists of Modernization 
Development Programs (MDEPs), or in some cases, increments of 
MDEPs. In this way, the decision maker will assess the 


contribution of each MDEP to the goal of Future Army 
Modernization. 
b. Mathematical Optimization 
In the second phase of the RDA3 decision support 
system, the values of the modernization initiatives (MDEPs), 
constrained by the annual funding proposal, are mathematically 
optimized. The mathematical optimization incorporates several 


decision maker constraints and goals such as annual budgetary 


limitations, target mission funding levels, and funding of 
Congressionally or DOD mandated programs. The mathematical 
Speimization currently in use is a new, enhanced math 
programming algorithm developed by Captain Scott Donahue, USA, 
in his thesis, while a student at the Naval Postgraduate 
School under the direction of Professor Rosenthal. 
2. Advantages of RDA3 

The decision support system, RDA3, possesses many 
characteristics which contribute to its value as a decision 
support system. Specifically, RDA3 


® is a comprehensive decision support system that considers 
every MDEP a potential modernization candidate. 


@® can be executed quickly, with an estimated time of just 
two to three hours for analysts to examine the outputs 
received from one iteration and formulate revised inputs 
that reflect new options to be explored. 


@® would remain applicable if a new math programming 
optimization technique was adopted, or if a new method of 
determining the hierarchal structure was implemented. 

® is versatile in that it will provide optimal solutions to 
budget decisions whether increasing expenditures or 
reducing spending authorization. 


® is highly transportable since it is built primarily on the 
GAMS linear programming language. 


@® is used to derive relative values or priorities. 


® is controlled by HQDA which provides projected research, 
development and acquisition (RDA) constraints. 


The many advantages of RDA3 make the system of great value to 


the Army leadership. The system also has the advantage of 


flexibility in the area in which this thesis recommends 
change: the hierarchal structure. 
3. Modernization Initiatives 
Factors that influence the overall desirability of 
modernization initiatives: 
® modernization plans, 
e field experience, 
e attitudes in the Office of the Secretary of Defense, 
® cost, 
e Congressional attitudes, 
e warfighting contribution, 
e history, 
e business sense, 


®e personnel considerations. 


These factors, however, are not necessarily an exhaustive 
list, and there is nothing to prevent the decision maker from 
reducing the influence of the above factors, or even 
eliminating one or more factors from consideration entirely. 

Given the above factors, TRAC decided the best 
quantitative scale and measurement process that captures the 
essence of these influences is the Analytic Hierarchy Process 


(AHP) . 


II. THE ANALYTIC HIERARCHY PROCESS 


A. SOURCE 

The background information in this chapter on the Analytic 
Hierarchy Process (AHP) is mainly drawn form the creator of 
AHP, Dr. Thomas L. Saaty. Specifically, the article that 
provides a description of how to use AHP, and an example of 
using AHP to buy a house are from "How to make a decision: 


The Analytic Hierarchy Process" [Ref. 1]. 


B. HOW TO STRUCTURE A DECISION PROBLEM 
Arrange the factors that are important for the decision in 

a hierarchic structure descending from an overall goal to 
criteria, subcriteria and alternatives. Include enough detail 
© ; 

® thoroughly represent the problem, 

® consider environment of problem, 

® identify issues or attributes, 


® identify participants. 


The arrangement of a hierarchy serves two purposes: Iie 
gives an overall view of the relationship and helps the 
decision maker determine if the issues in each level are in 
the same category so he can make accurate comparisons. Verbal 


judgements are numerically given values of (1,3,5,7,9) ranging 


from equal to extreme (equal, moderately more, strongly more, 
very strongly more, extremely more). These values are the 
result of comparing the more preferred alternative to the 
lesser preferred alternative. When comparing a lesser 
preferred alternative to a more preferred alternative, as 


expected, the numerical values would be the reciprocals of the 


above, or (l,l,i+2,¢). The aspect of judgements is 
discussed in the next section. 


C. PAIRWISE COMPARISON SCALING 

A primary concern of AHP is with scaling. Saaty states 
"The Analytic Hierarchy Process 1S rigorously concerned with 
the scaling problem and what sort of numbers to use, and how 
to correctly combine the priorities resulting from them" [Ref. 
1:p. 10]. The table Saaty devised to use when making pairwise 


comparisons is shown in Figure 1 [Ref. 1l:p. 15]. 
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Experience and 
Judgement 
strongly (sic) 
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The evidence 
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ye4a,6,8 Intermediate When compromise 


values between the | 1s needed 
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two adjacent 
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Rationals Ratios arising If consistency 
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If activity i has one of the above 
numbers assigned to it when compared 
with activity j, then j has the 
reciprocal value when compared with i 





obtaining n num- 
erical values to 
span the matrix 


Figure 1. The AHP Point Scale for Pairwise Comparisons. 


D. PAIRWISE COMPARISONS 

The aspect of pairwise comparisons that is described below 
assumes a decision maker is evaluating a problem of m 
attributes and n alternatives using AHP. 

The attributes that contribute to the value of each 


alternative to be evaluated (Saaty refers to the attributes as 


"Criteria"), are evaluated by pairwise comparison by the 
decision maker. In an assessment of m attributes, the 
decision maker is required to perform “72 pair 
comparisons. The decision maker would construct a pairwise 


comparison matrix with m rows and m columns where each row and 


column represent an attribute. The number of pairwise 


comparisons is reduced from m* by two requirements of AHP. 





First, the matrix will be reciproead. That 1s each element 
r;; of the pairwise comparison matrix is the reciproca iia 
On 

r.4=— Vi a7 m 

a7 r red fe @ @ ¥ e 


Second, main diagonal elements of the matrix will always have 


entries of one. Rheaite is 


The pairwise comparison procedure is performed at every 
level of the hierarchy, with the exception of the alternatives 
themselves, the lowest level of the hierarchy. lie Ouner 
words, if there were one or more levels of subcriteria, 
pairwise comparisons in the manner described above would be 
performed. 

In the final or lowest level of the hierarchy, the 
decision maker would make pairwise comparisons of alternatives 
one attribute at a time. The decision maker would evaluate 
each alternative by attribute. Ultimately, the decision maker 


will have m matrices (one for each attribute) of size nxn. 


n(n-1) 


Bach matrix is formed by making pairwise comparisons of 


the alternatives. 
The pairwise comparison matrices are said to be consistent 


if there 1S a vector w of size n, in the case of alternatives 





(Ww would be of size m in the case of attributes), such that 
W . ae 
Ti; 2 Wee Ga 1, r r ae ae (1) 
G) | 
J 
Otherwise, the matrix 1S not consistent. Note that these 


equations imply that 


Diy ty tee ee (2) 


for consistency. The vector w is made unique by normalizing 


by dividing by its sum. Thus, 


Nn 
: @,=1. 
1=1 


If we refer to the matrix of pairwise comparisons as R; R 


is consistent if, and only if 
R@®=nNo. 


In a problem where there is some inconsistency present, AHP 


solves 
RO=)_,,@ 


where A,., is the principal eigenvalue of R. This leads to an 


x 
approximation of w whose entries correspond to the weights of 
the alternatives or attributes. To determine the amount of 
inconsistency and determine if the amount of consistency is 
acceptable, Saaty developed the consistency index (CI) defined 


as 


(n-1) 
Saaty recommends if CI < 0.1, accept the estimate of w. 
Otherwise, attempt to improve consistency. He does not 


specifically state what course of action the decision maker 


should undertake to improve consistency [Ref. l:p. 13). 
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The decision maker could be forced into consistency by 
making just n pairwise comparisons in the first row of the 
pairwise comparison matrix. The first pairwise comparison 
would be alternative one compared to itself, which is by 
definition unity. Excluding this comparison, (n-1) pairwise 
comparisons are all that are necessary. In this way, the 
decision maker would obtain the entries for the first row of 
the pairwise comparison matrix, and define the weights based 


on those entries 
ene ois ae ee =Mear, Woo. eee 7 wn) 


The entries for the rest of the matrix could be obtained by 





defining 
W ; 
Ii5- 2 he Fy — ls » . . Paes 
G) | 
j 
and again normalizing weights to sum to one. With these 


weights obtained exclusively from the first row the pairwise 


comparison matrix, every element of the matrix could be 


obtained. The resulting matrix R would be perfectly 
consistent. 

However, AHP, to its credit, does not force this 
consistency on the decision maker. By requiring +2 


comparisons, AHP makes the vector w over-determined and allows 


inconsistencies. 


Wb 


Therefore, the pairwise comparison matrix Coulamivem, 
likely contain inconsistencies. With the presence of 
inconsistencies, there is no exact solution for the vector @ 
such that Equation (1) holds for every 1 andj. The question 
is how to find an w that "best" fits these equations when 
inconsistency is present. With little justification, AHP uses 
the eigenvalue method previously described. The advantages of 
this method are (1) if the pairwise comparison matrix is 


consistent, A,,,=n, and (ii) it allows evaluation “oF 


consistency by the consistency index (CI) defined above. 

In Chapter IV we discuss and evaluate other methods of 
determining the vector w. All methods produce exact results 
if the decision maker is consistent. We evaluate 
approximations by defining the error between the pairwise 
comparison matrix and one produced from the vector of weights, 


Ww. 


E. EXAMPLE 

In order to illustrate how to use AHP in a problem, 
attached as Appendix A to this thesis is the example of 
choosing the best house to buy using AHP from Saaty’s article 


cited above [Ref. 1]. 


Pr. INSIGHTS 
In light of this chapter’s discussion concerning @anee 


several questions may have occurred to the reader. Fiom 
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example, given the AHP system of pairwise comparison scaling, 
how does the decision maker rate three alternatives if the 
first is judged to be of extreme importance to the second, and 
the second is determined to be of extreme importance to the 


third? There is no 81 on the scale, which is what is required 
Pert =) sao and Equations (1) and (2) have to hold. 
Additionally, how does the restriction to an integer scale 


compromise the accuracy of pairwise comparisons? Consider 


three alternatives which result in the following pairwise 


comparisons. The first is preferred by a factor of three to 
the second (1r,,=3). The third is preferred by a factor of two 
to the second (2,,=2). By how much is the first alternative 
preferred to the third? Equation (2) shows that for 


@emsistency to hold, 7,,=1.5, but that is not on the AHP scale 


and so could not have been chosen. 
These problems and others will be discussed thoroughly in 


the next chapter. 


des 


III. CRITIQUE OF THE ANALYTIC HIERARCHY PROCESS 


A. BACKGROUND 
The Analytic Hierarchy Process contains several flaws, 
three of which are discussed in this chapter. Two issues, 
scaling and ordering, are significant concerns, yet the 
problem of rank reversal represents a flaw in methodology and 
is considered the most significant issue regarding the 
usefulness of AHP. It is for this reason that the problem of 
rank reversal must be viewed as far more significant than any 
of the other problems. The result is that AHP can produce 
rank orderings that are not consistent with the underlying 
preferences of the decision maker. In Chapter IV, we show how 
to avoid this while maintaining the ratio scale. 
1. Rank Reversal 
By adding or deleting an alternative, reversal in the 
ranking of the other alternatives can occur, even though this 
addition or deletion causes no change in the decision maker’s 
pairwise comparisons. As ironic as it may seem, AHP can tell 
a decision maker that his or her preference of alternatives 
would change if an additional alternative were to be 
considered. Even if the new item considered has the same 
attribute levels as an original alternative, AHP could tell 


the decision maker to change original preferences. This is 
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what is described as rank reversal. The concept will be 
covered in greater depth later in this chapter. This has been 
much discussed in the literature. See, for example, Schoner 
and Wedley [Ref. 2], Howard [Ref. 3], Holder [Ref. 4], Roper 
and Sharp [Ref. 5] and Belton and Gear [Ref. 6]. Saaty 
Maintains that rank reversals are justified because one rates 
the uniqueness of an alternative. The presence of a duplicate 
of this alternative must, he claims, decrease the value of the 
Original alternative, even if the decision maker says it does 
not affect his or her relative strengths of preferences. 

2. Pairwise Comparison Scaling 

Pairwise comparisons are restricted to a nine point 

integer scale. This necessarily leads to inconsistency since 
no attribute or alternative can be 1.5, 4.5 or 10 times more 
important than another. 

3. Weight Fitting 


The most appropriate way to determine the weights o, 


from pairwise comparisons may not be the eigenvector method 
used by AHP. In an article by Hihn and Johnson [Ref. 7], 16 
methods are studied, and there appears to be no reason to 
believe AHP generates the most desirable solutions using the 
elgenvector technique. 

Now we move to a discussion of how AHP is used in 
practice and how AHP could be changed in a fundamental way. 


In a problem in which AHP is used, let us assume that there 


AL'S 


are n attributes indexed i=1,2,...,n with the weight of 


attribute 1 given the value a,. The attributes watt wae 


normalized so that 


We would like to determine a Meet. eta of weights 





@W=(W,,@5,...,@,)7 that denotes positive weights obtained by 
the pairwise comparisons. We will define 
ee 


This represents the ratio of the weights of attributes 1 and 
Jj which, in Saaty’s method, is determined through pairwise 
comparisons. As was pointed out in Equation (2), if the 


decision maker is consistent, the ratios will satisfy, 











VoL oe 


If there is inconsistency, determining the "best" w is a 


nontrivial matter. There are n @,;’'s to be found using 


oe r;;'S, so the problem is how to best £1t the @) (oa 


an over-determined data set. 
In AHP, the matrix of these ratios, or comparisons, 
will approximate the vector w of the weights by solving the 


following equation 


16 


Iii Fiz Fin || 1 W, 
Io, F22 Fon| | @2] _ W > ea 
= 
c 5 max i 
Dn Tne fa. Tn WO, W > 


where the matrix of ratios (data from pairwise comparisons) 
will be labelled R. Equation (3) can be expressed in matrix 


notation as shown below: 


Characteristics of matrix R include 





The diagonal entries of R are all one, since a pairwise 
comparison of an attribute with itself must be one. 


Saaty’s method forms the matrix R by performing 


n(n-1) 


; pairwise comparisons. The next step is to determine 


the largest eigenvector of R and the corresponding right 
eigenvector. This vector is normalized to give the relative 


weights. Let A,., represent the largest eigenvalue of R. We 


list the following results found in the AHP literature: 
® R is reciprocal, 


® R is not necessarily consistent, 


ay, 


® R is consistent if and only if 4237 


In AHP, the nine point scale, as previously described, is used 


to make comparisons. Saaty advises if 


use the results obtained, since comparisons are "consistent 


enough." However, if 


identify the inconsistencies, modify the comparisons and 
recalculate. This appears to be a rule-of-thumb with no 


theoretical foundation. 


B. RANK REVERSALS 

An axiom of rational decision making implies that addition 
or deletion of a new alternative, with no changes in relative 
preferences for existing alternatives, should never cause a 


change in the existing ranking when ranking alternatives. 


Saaty argues "rank reversals do occur in practice in a way 
that does not satisfy these assumptions" [Ref. 8:p. 13]. In 
way of an explanation, Saaty continues, "On seeing too many 


copies of an attractive alternative, one abandons it for a 
less attractive one that is unique" [Ref. 8:p. 13]. In an 


example of rank reversal later in this section, rank reversal 
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of three tanks occurs when a duplicate of one of the tanks is 
added. Presumably, the decision maker is not planning on 
fighting a war with one tank and is aware that there are other 
tanks in existence of all types of tanks considered. In fact, 
it could be argued that in this case uniqueness of an 
alternative would be a weakness. In this case, however, AHP 
rates an alternative as less preferable when a duplicate of 
that alternative is considered despite the fact that 
uniqueness is clearly not an advantage. 

Saaty shows the frequency of rank preservation of all 
alternatives for 1000 random cases involving two to nine 
criteria and two to nine alternatives in distributive, ideal 
and utility modes of AHP [Ref. 8). In the distributive mode, 
weights derived from paired comparisons are normalized to add 
to 1.0. The ideal mode divides by the weight of the highest 
ranked alternative for each criterion making the largest 
weight to be 1.0. The utility mode is a transformation of the 
ideal mode to an interval scale. The trend in all three cases 
shows the rate of rank preservation decreases as number of 
criteria and alternatives increase. In other words, the more 
complex the problem in terms of criteria and alternatives, the 
more likely it is that one will have rank reversal. In the 
distributive mode with two criteria and two alternatives, 
there were 963 cases out of 1000 of rank preservation. When 
nine criteria and nine alternatives were used, rank was 


preserved just 526 times out of the 1000 cases. Similarly for 


a) 


the ideal mode, 967 cases of rank preservation for two 
criteria and two alternatives reduced to 832 cases with nine 
alternatives and nine criteria. Finally for the utility mode, 
903 cases of rank preservation with two criteria and two 
alternatives were reduced to just 664 cases of rank 
preservation when nine criteria and nine alternatives were 
used [Ref. 8]. 
1. Duplication 

In 1983, shortly after the introduction of AHP, itive 
shown that the introduction of a duplicate alternative could, 
in some cases, cause a rank reversal of the other alternatives 
fRef. 6].! Saaty responded to this by arguing that rank 
reversals are justified because an alternative loses some of 
its appeal if it is not unique. In reality, sometimes a 
duplicate may reduce the value of an alternative, but in AHP, 
Guplicates of alternatives always reduce the value of the 
original alternatives. The decision maker has no choice. Vi 
a duplicate alternative is introduced into the decision 
problem, the value of the original alternative is reduced. 
The rank reversal will still occur even if the uniquenéssmam 
an alternative is irrelevant to the decision maker. 

This problem is caused by the particular normalizing 


of weights in AHP. This is inherent to AHP and cannot be 


The rank reversal example of Belton and Gear is reproduced by 
Schoner and Wedley in Reference 2. 
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eliminated without changing the way AHP normalizes weights, 
which is exactly what we hope the distributive method has 
done, but the article [Ref. 8] fails to explain the source of 
the rank reversals. The 474 cases of rank reversal out of the 
1000 cases in which nine criteria and nine alternatives were 
used with the distributive method could be due to 
inconsistency in pairwise comparisons, or renormalizing 
weights without regard to units of measurement. We can not be 
sure if the distributive method is a viable alternative to 
AHP. There can be two sources of rank reversal. The first 
source iS inconsistency in pairwise comparisons by the 
decision maker. The second source is from normalizing 
attributes of alternatives in away that does not maintain a 
ratio scale. The former source of rank reversal does not 
appear to have a solution. The latter source can be solved by 
elther of the modifications described in this thesis. We must 
caution that Saaty’s distributive method might cause many or 
all of the rank reversals that occurred by a flawed method of 
mermalizing. 
2. Near Copies 

In response to rank reversals upon the introduction of 
Guplicate alternatives, Saaty, rather than recognize this 
issue aS a potential source of error in the methodology of 
AHP, suggested elimination of alternatives from consideration 


that score within 10 percent of another alternative. In an 


fee 


example later in this chapter, the additional alternative 
introduced, the third tank, is not within 10 percent ef eens 
of the first two tanks in either of the two attributes 
evaluated, yet a rank reversal still occurs. 
3. Rank Reversal Example 
What follows are examples of how the Analytic 
Hierarchy Process (AHP) can produce a rank reversal in 
alternatives when an additional alternative is added. It also 
follows that a rank reversal can occur when an alternative is 
eliminated from consideration. These examples simulate making 
a decision on selecting or valuing a weapon system, in this 
case a tank. However, much like the example of using AHP to 
compare houses (which is in Appendix A to this thesis), this 
example 1s intended to be generic and egually applicable to 
all ranking or valuing problems. Therefore, the problem of 
rank reversal is not a special case or exception when AHP is 
used. 
a. Attributes 
Assume that there are two attributes, survivability 
and firepower. Further assume that you decide that firepower 


is 1.25 times as important as survivability.* Weights are 


(2,2) for survivability and firepower, respectively. This 


“Although the literature on AHP calls for using the scale in 
Figure 1, the computer application of AHP, "Expert Choice" aes 
allow for a continuous set of comparison numbers to be used. 
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maintains the weight Piancitetes (iazox— —— ), and meets the AHP 


requirement that weights sum to one. The example of how these 


attributes satisfy the equality of Equation (3) is shown below 


where A,.,=n=2 


S F 
| Se ie a 
5 9 9 

: ples |e 
2) > > 
Pe\ 4 9 9 


The labels S and F represent Survivability and firepower, 
respectively. The matrix 1S consistent, as it must be in any 
2x2 reciprocal matrix with entries of one on the main 
diagonal. 

(1) Survivability. On comparing three types of 
armor, reactive armor is preferred 1.5 over applique, and 
applique is preferred 2 over rolled homogeneous armor. 

(2) Firepower. A tank with a 130mm main gun is 
preferred 1.4 over a tank with a 120mm main gun. A 120mm main 
gun is preferred 1.5 over a 105mm main gun. A 105mm main gun 
is preferred 2.0 over a 90mm main gun. 

b. Comparing two Tanks using AHP 

Tank 1 has applique armor and a 120mm main gun. 

Tank 2 has reactive armor and a 105mm main gun. So Tank 1 has 


more firepower but is less survivable than Tank 2. 


Zs 


The consistent matrix equation satisfying Equation 
(3) from the pairwise comparisons of Tank 1 with Tank 2 for 


the attribute of survivability is shown below: 


Ie, 2% 
7T71/ 4 4 mic md 
S 10 =e LO 
Zz 10 16 


Similarly, the matrix equation satisfying Equation (3) @iimem 
the pairwise comparisons of Tank 1 with Tank 2 for the 


attribute of firepower is shown below: 


T1 T2 
T1211 2/) & & 
2|| 10) _,| 10 

es a 10 10 


Figure 2 shows the method used to determine 
relative value of alternatives once the weights of attributes 


have been assigned and pairwise comparisons of alternatives 


have been made. 


{+-—___-— ATTRIBUTES RELATIVE 


SURVIVABILITY BIRR POWER 
(4/9) (S73 


VALUE 





Figure 2. Comparison of two tanks using AHP. 
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The figure shows Tank 1 is preferred to Tank 2, or 


Tank | > Tank 2. 


If we let R,, be the pairwise comparison matrix 


obtained by pairwise comparisons of the relative value of the 


tanks, we obtain the following matrix: 


ia T2 
-T1/ 1 1.045 


R = 
ee T? | 07957 1 


c. Additional Alternative 
Tank 1 has applique armor and a 120mm main gun, and 


Tank 2 has reactive armor and a 105mm main gun. These are the 


same tanks aS in the section above. Tank 3 has rolled 
homogeneous armor and a 130mm main gun. Tank 2 has the most 
Survivability and Tank 3 has the most firepower. Tank 3 has 


the least Survivability and Tank 2 has the least firepower. 


tame. was rated between Tanks 2 and 3 in both attributes. 


ARP BOLT ES RELATIVE 


SUI NAO NE be FIREPOWER 
(4/9) (5/9) 











Figure 3. Comparison of three tanks using AHP. 


Zo 


When Tank 3 is included, Tank 2 becomes preferred 
to Tank 1 which is in turn preferred to Tank 3. sag m7 
shows Tank 2 is preferred to Tank 1, which is preferred to 


Tank “3, Or 


Tank 2 > Tank 1 > Vance 


Note that this reversal happened even though no changes were 
made to the relative importance of the options in either 


Survivability or firepower, or between these two attributes. 


If we let R,, be the pairwise comparison matrix obtained by 


pairwise comparisons of the relative values of the three 


tanks, we obtain the following matrix: 


fee i iG) 
ll i OOS 10k 


Re = 02 143 eet 
3 | 02.997 ©. 936 i 
If the ratio scale had been maintained, the pairwise 
comparisons between Tank 1 and Tank 2 should be the same as 


Pe 


they were in R,,. This is not the case. 


d. Explanation of Rank Reversal 
In the original formulation that compared two 
tanks, the dilution of components of relative value can be 
seen in a comparison of the two tables showing component 
relative values before and after Tank 3 was included. 


Originally, the component in the table contributing the 
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largest amount to either tank’s relative value is represented 
by the firepower attribute of Tank 1. This can be read off 
the table as 0.333. This 1s significantly higher than the 
relative value component of Survivability of Tank 2 which is 
only 0.2666. With the addition of Tank 3 which has a large 
rating for firepower, the relative value components of 
firepower for the two original tanks (Tank 1 and Tank 2) are 
Significantly reduced. Once Tank 3 is added as an 
alternative, all components of relative attributes of the 
Original two tanks are reduced. In this case, Tank 3 was 
rated higher in firepower than survivability. Therefore, the 
firepower rating of Tank 1 (originally 0.333 then 0.181 fora 
reduction of 0.152) had a greater reduction in magnitude than 
the survivability rating of Tank 2 (originally 0.2666 then 
0.222 for a reduction of only 0.0446). The greater reduction 
in magnitude of the firepower attribute of Tank 1 led to the 
rank reversal with Tank 2. 
e. Duplication of Alternative 

Now let us suppose a fourth tank is to be 
considered. The fourth tank has the same characteristics as 
Tank 2. Therefore, since it also has reactive armor, it will 
Mempretrerred by 3 to 1 over Tank 3 in survivability, which has 
rolled homogeneous armor. It is preferred by 1.5 to 1 over 
Tank 1 with survivability characteristic of applique type 


armor. Similarly in firepower, its small diameter main gun is 
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least preferred. Both attributes must be renormalized and 
multiplied by the weight of the attribute to obtain the new 


relative value of the tanks. This is done in Figure 4. 


ae ATTRIBUTES 
RELATIVE 
SURVIVABILITY FIREPOWER VALUE 
eer a 


Oe | 9.2222 PN eee Zou og | Oa Sal 2475 


TANK 2 Oe SS O32 O. 242 
TANK 3 0.3755 0.2581 


TANK 4 O oro Or 1 oe 0.2472 
Figure 4. Comparison of four tanks using AHP. 





The decision maker might consider the uniqueness of 
a tank to be irrelevant, but nevertheless, AHP produces a 
dramatic rank reversal in this case. Recall that originally 
AHP produced a preference order of two tanks as Tank 1 > Tank 
2. With three tanks, the order becomes Tank 2 > Tank 1 > Tank 
3. Now by adding a fourth tank with the same characteristics 
as Tank 2, the preference of the four tanks is 3, 1, 2, 4, 
that is Tank 3 > Tank 1 > Tank 2 = Tank 4 (the "worst" becomes 
the "best")) . 

f. Discussion of Results 

The reversal in preference order of the first two 
tanks by the addition of a third is caused by flawed logic in 
AHP and the disregard for measurement units. Even though 
there are no natural units for meaSuring survivability and 


firepower, one must define an underlying scale based on a 
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Standard. Both the scale intervals and standard can be 
arbitrarily chosen, but once they are, measurements must be 


mace Consistent with this scale. 


C. PAIRWISE COMPARISON SCALING 

There is inherent inconsistency in the nine point scale 
used by AHP. The nine point pairwise comparison scale is 
defined in Figure 1 in Chapter II. 

Immediately, it is possible to pose theoretical problems 
impossible to formulate when restricted to this nine point 
scale. This is the case if one is faced with three 
alternatives, A, B, and C with the following pairwise 
comparison results. Alternative C is deemed to have extreme 
importance over B, and B 1s deemed to have extreme importance 
over A. One would assume C would have much more than extreme 
importance over A, but this is not possible with AHP. To say 
that alternative C is just of extreme importance over A defies 
Magic. That would imply that the decision maker should be 
indifferent between B and A (after all, C is of extreme 
importance over both these alternatives). Perhaps the 
decision maker should now decide alternative C now only has 
strong importance to B and B only has strong importance to A 
so C can have extreme importance over A. This new hierarchy 
is neither consistent, nor does it explain why a decision 
maker should revise his or her estimate of the pairwise 


comparison of alternatives. 
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The situation above was entirely theoretical with generic 
alternatives. But what if the problem was travel from the 
west coast to the east coast, and alternative A was walking, 
alternative B was driving a car and alternative C was flying 
a plane. It seems obvious alternative C would be preferred by 
more than a factor of nine to alternative A. 

Inconsistency is not the exception with this nine point 
scale. Almost any time there are more than two or three 
comparisons to be made, the scale will almost always lead to 
inconsistencies. To maintain consistency, the pairwise 
comparisons of alternatives should be the product of the scale 
number determined. Take, for example, four alternatives, A, 
B, C, and D, that are deemed of moderate importance over their 
preceding alternative. Therefore, alternative A, being the 


least preferred alternative, would receive a 1 in a pairwise 


comparison to itself. Alternative B would receive a 3 in 
comparison to A (moderately preferred). Similavaaw 
alternative C receives a score of 3 compared to B, or 9. But 


what score does D receive? It 1s moderately preferred to C, 
but C already has the highest score allowed on the nine point 
scale. 

In theory, there should be no limit to the scale. The 
scale should be open ended, and not restricted to a maximum 
value of nine. 

One simple way to avoid the scaling problems described 


above would by to insure the decision maker stayed within the 
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scale. The decision maker would identify the extreme 
attributes, that is the most and least preferred attributes. 
The least preferred attribute would be defined as one on the 
scale, and the most preferred would be defined as nine. All 
other attributes would, therefore, fall somewhere in between 
these two extremes and all attributes would be within the 


scale. 


D. WEIGHT FITTING 


In any evaluation of attributes that allows inconsistency, 
the user must somehow be able to determine how much 
inconsistency there 1s in the evaluation. Given a matrix R = 


[r,,], suppose the weights w have been obtained. One 


expression that measures the error between the weights and the 


meaeios is 


A second and more tractable error measure 1s 
ya ia 5 
Brror= & 4S (@.7,,-w,)*. (4) 
2=) 27 =1 
Both of these expresSions are zero if, and only if, R is 
consistent. One definition of the "best" weights w is that 
they minimize the expression in Equation (4) for a given 


matrix R. This is the method of least squared error (LSE) for 


evil 


evaluation comparisons, and was the first method we selected 
to evaluate preference fitting. Without further comment, we 
point out that the above equation is not the unique way to 
evaluate consistency. In fact, the article by Hihn wage 
Johnson [Ref. 7] lists and describes 16 methods of finding the 
"best" w for a given R for various definitions of "best" ee 
appears that none of the methods differ from each other a 
great deal, but Saaty’s method was one of the poorest. 
Saaty’s method, the second method we evaluated by using 


preference fitting, is to solve 


where A,., represents the largest eigenvalue of the matrix R. 


x 


In all cases, A,,, 2=n, and when R is consistent, A,., = Nn. 


Hihn and Johnson point out that Saaty’s justification Siem 
selecting this particular method, the right eigenvector 
technique, is that if Ris consistent, this technique produces 


the solution A,,, =n. But they also point out that all of the 


x 
16 methods discussed in Reference 7 produce this exact 
solution in the case of consistency. Therefore, the “omik 
method of differentiating the 16 techniques is to evaluate 
each technique’s results when inconsistency is present. Hihn 
and Johnson show that Saaty’s technique is one of the poorest 


of the 16 techniques evaluated. 
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The third and final method we studied in the area of 
meeterence fitting we call the dimension method. In this 


method, we solve the following equation 
R® =n. 


1. Approach 
In each of the three methods, the error matrix was 
evaluated based on the LSE formula. This gave a value for the 
total error of the method used. The value of this total error 
does not necessarily prove any one method is better than 
another. We did, however, obtain interesting results using 
Saaty’s pairwise comparison matrices from his example of using 
AHP to buy a house [Ref. 1]. 
2. Results 
A 3x3 matrix of attributes was evaluated using Saaty’s 
right eigenvector method, the LSE method and what we call the 
dimension method. The matrix is reproduced below with results 
of pairwise comparisons as entries. As required, each main 


diagonal entry is a one, and the matrix 1S reciprocal. 


1 6 
1 6 8 
Peer 0.17. 1. “4 |, >) 
aah Os Ora eel 
8 4 


Using Saaty’s method, the priority vector of weights was 
Seeermminead as (0.754,0.181,0.065)7 with an error of 17.82 


(Anax=3-136). We will denote the matrix obtained by AHP as 


38 


Reaary- The result using least squares was a priority vector 


of weights of (0.727,0.202,0.071)7% and error ore eee 
will denote the matrix obtained by the LSE method as R,g,. 


Finally, the dimension method produced a priority vector of 
weights of (0.768,0.145,0.087)? with an error of 12.) ee 
will denote the matrix obtained by the dimension method as 


aay - We point this out not as an evaluation of any 


technique, but simply to demonstrate that different techniques 
produce different results. Naturally, the LSE method had the 
smallest error. Next was the dimension method, and finally, 
with the highest error of any of the three methods, was 
Saaty’s elgenvalue method. The reader should compare the 
following three matrices with the one in Equation (5) and 
decide which of these is "closest" to R. Note that, as 
expected, the LSE method fits the larger numbers in R better 
than the smaller ones. 
a AY, 16 dG i 3,607 10% 4 


0.24 12s Rie. = | 0°28) ae 
0.09 0.36 1 0.10 0.35 see 


Reaary = 


dk S29) Vee 
Ones il 2.84 
Or dy OG M3 


DIM ~ 


These results were from using these methods in one of 


the pairwise comparison matrices for the three alternatives in 
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just one of the eight attributes considered. We were curious 
to see how each method compared if used in the overall 
problem. The results of an APL program used to evaluate 
errors by method for alternative pairwise comparison matrices 
are in Appendix C to this thesis. Fach of the eight 3x3 
matrices represents one of the attributes considered. Each 
matrix was used to determine a vector w of weights for the 
alternatives and each method was evaluated for the error term. 
Naturally, since the criterion used 1S minimum least squares, 
the LSE method had the smallest error. In all cases except 
one, the dimension method had the next smallest error. era 
seven of the eight cases, Saaty’s eigenvalue method had the 
largest error. 

The program in Appendix C also includes the 8x8 
pairwise comparison matrix to approximate the vector w of 
weights for each attribute, what Saaty refers to as level one 
of the hierarchy. The error term for this matrix showed the 
LSE method produced an error of 237.75. Saaty'’s eigenvalue 
method gave the next smallest error at 259.06, and the 


dimension method had by far the highest error term of 858.40. 
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IV. SUGGESTED MODIFICATIONS TO AHP 


A. DESCRIPTION OF MODIFICATION 

The two significant problems with AHP identified in the 
previous chapter, rank reversals and scaling, are addressed in 
this chapter. The modification we recommend for AHP is one 
that will avoid rank reversal. It can also avoid the problem 
of AHP’s nine point scale. 

1. Ratio Modification 

The decision maker, through pairwise comparisons, 

forms a matrix of ratios of pairwise %eompartsencs If thame 
matrix changes when additional alternatives are considered, of 
course rank reversals can be expected. If the decision maker 
elects to reassess his or her previous alternatives’ values, 
this 1S a separate matter entirely. Without the decision 
maker changing his or her original preferences, the ranking 
(and matrix of ratios) should remain unchanged. 

2. Scale Modification 

The most extreme comparison of alternatives should be 

used to establish the scaling for other alternatives. There 
is nothing wrong with the original nine point scale, or a 
scale from 1-10, but all numbers between one and the scale 
extreme would have to be included. Lacking this modification, 


the scale should be open ended to allow comparisons of 
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alternatives differing by orders of magnitude in value to the 
decision maker. 

Recall that AHP requires two types of comparisons and 
normalization of weights assigned. The Eimsk: Companitsion 
involves comparing the importance of one attribute with 
another. This phase of implementing AHP is unchanged by this 
modification. The other type of comparison is to determine 
the weight of each alternative’s characteristics by attribute. 
This phase of AHP should be modified so that the least 
preferred alternative in an attribute is assigned a value of 
one (or the most preferred). By doing this, the renormalized 
weights obtained by pairwise comparison of attributes will 
maintain a ratio scale and avoid rank reversals. 

3. Weight Fitting 

There seems to be no credible reason not to use the 
least square error (LSE) method to fit weights to alternatives 
or attributes. The examples in Hihn and Johnson [Ref. 7] are 
convincing. The eigenvalue method used by AHP is mediocre, at 
best, among the 16 techniques evaluated for weight fitting. 
The justification for use of the eigenvalue method in AHP is 
that an exact solution is obtained in the case of consistency. 
But exact solutions are produced by all 16 techniques in the 


case of consistency. 


So 


B. REASON FOR MODIFICATION 

The Analytic Hierarchy Process (AHP) is currently used to 
obtain coefficients of the optimization program for the math 
programming algorithm used by the Director, Program Analysis 
and Evaluation to maximize future Army modernization subject 
tO a2 L£UNnGING ‘cConeetra inate 

To assess the desirability of a management decision 
package (MDEP), TRAC decided to use the AHP as a quantitative 
scale and measurement process. This thesis shows that AHP 
will provide coefficients for the optimization algorithm that 
are not measurable and can be subject to rank reversal if 
certain alternatives are added or deleted from the original 
Formulation. Even if rank reversal does not occur, how can 
one have any confidence in the ranking measures when they 
change due to no changes in the preferences of the user. The 
idea of using such a flawed system for making major investment 
decisions is very disturbing. This thesis determines a way 
to include some aspects of AHP, but not suffer from rank 


reversal problems. 


C. APPROACH 

Each MDEP can be converted into its relative contribution 
to Future Army Modernization through multiple levels of 
hierarchy. Although the MDEP’s contribution to Future wae 
Modernization may be a meaningless value to a decision maker, 


it will be consistent in its relationship to other MDEPs 
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because all MDEPs will be expressed in the same "units" of 
Future Army Modernization. Additionally, if it is necessary 
to add new alternatives, the original alternatives will 
maintain the same values of coefficients for the math 
programming algorithm. 

This procedure is consistent for multiple level problems. 
The alternatives will receive the same value whether the 
problem is solved with all levels used, or if the problem is 
reduced to a single level by cross multiplying weights for 
each level. 

This thesis shows that any model of alternatives and 
attributes can be expanded to include other attributes without 
encountering any problem with rank reversal. 

1. Additional Attributes 

The weight of each attribute will remain the same. 
Elements of each attribute will not be renormalized to sum to 
one if additional alternatives are introduced, or if original 
alternatives are eliminated. In fact, even in the original 
formulation, weights of alternatives by attribute will not be 
normalized to sum to one. This implies that the value of each 
Original alternative will not change because the value of an 
alternative is the weight of each attribute times the value of 
the alternative’s attribute summed over all attributes. The 
relative value of alternatives should not change if more or 


eas alternatives are considered. 
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In the original formulation, all altérnatives wis, 
attribute will be scaled by one of two methods. First, the 
decision maker can assign a value of one to the most preferred 
alternative for one attribute. We call this the "standard 
attribute." In other attributes, alternatives are assigned a 
value based on the decision maker’s value comparisons between 
attributes. The decision maker must create a link between 
attributes by declaring an equivalency between an alternative 
of one attribute and an alternative in other attributes. One 
can not arbitrarily decide that the decision maker places 
equal value on all most preferred alternatives in every 
attribute and assign each a value of one. The most preferred 
alternative in other attributes could be "better" (assigned a 
value greater than one) or "worse" (assigned a value of less 
than one) when compared to the decision maker’s "standard 
attribute" value of one. New alternatives are simply assigned 
values based on the original formulation’s value in each 
attribute. If the new attribute is preferred by a factor of 
three to the "standard attribute" from the original 
formulation, it will be assigned a value of three. If the new 
alternative’s attribute is less preferred than the original 
formulation’s most preferred attribute, the (0,1) scale will 
be maintained. Even in the original formulation, however, the 
elements of other attributes may not fit a (0,1) scale. 

When using the second method, the decision maker can 


assign a value of one to the lowest or least preferred 
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alternative for one attribute, which we will refer to as the 
Metandara attribute." In this attribute, new alternatives are 
simply assigned values based on the original formulation’s 
least preferred attribute values of one. In other attributes, 
alternatives are assigned a value based on the decision 
maker’s value comparisons between attributes. 
Zoe applicability 

The modification to AHP discussed in this proposal 
would be a Significant improvement to the method used by PAE 
to obtain coefficients to the math programming algorithm to 
optimize future Army modernization. This method would insure 
no rank reversals in alternatives whether alternatives are 
duplicated, eliminated, or completely new alternatives are 
added. In addition, the values of each alternative will have 


consistent units that can allow comparisons of magnitude. 


D. THE TANK PROBLEM REVISITED 
1. Description of Approaches 
This section demonstrates the use of our suggested 
modification to AHP in two ways. Both avoid rank reversals 
that plagued our earlier examples. 
a. Link between Attributes 
To establish and maintain a ratio scale and avoid 
rank reversals, the decision maker must compare the elements 
of different attributes and determine a relative value between 


a specific element of one attribute and an element of other 
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attributes. It is essential to have a link between 
attributes. Otherwise, an alternative with an extremely 
undesirable attribute would give overwhelming value to other 
alternatives simply by assigning the least preferred 


alternative a value of one. We assume the decision maker, 


given the weights of (],2) for survivability and firepower, 


respectively, assigns the same value to survivability of 
applique armor and firepower of a 105mm main gun. We 
illustrate this concept by a simple example. 

We assume the decision maker has a tank with 
attributes of applique armor and a 105mm main gun, and could 
improve attributes. His or her priority for doing so would be 
equal to the weights of the attributes. That is, the decision 
maker would have preference for improving firepower of 1.25 
times his or her preference for improving survivability. This 
illustrates a serious flaw with AHP. 

The weights of attributes cannot be independent of 
alternatives. In our case, would the decision maker have the 
Same priorities given a tank with rolled homogeneous armor and 
a 130mm main gun? The decision maker possessing a tank with 
the lowest survivability attribute (rolled homogeneous armor) 
and the highest firepower attribute (130mm main gun) should be 
more concerned about improving survivability. 

In this modification, the survivability )raueae 


could be tripled by replacing the rolled homogeneous armor 
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with reactive armor. This will contribute = of the 


Survivability improvement to the tank’s relative value. 


Although the firepower attribute will contribute 2 of its 


improvement to the relative value of the tank, the firepower 
rating is already the highest obtainable. It would almost 
certainly be nearly impossible to triple the firepower rating. 
Therefore, improving the survivability attribute would 
contribute a greater increase to the relative value of the 
tank than increasing firepower. Despite the lower weight for 
the survivability attribute, a decision maker will elect to 
improve survivability over firepower if he or she possessed a 
tank with a high firepower rating and low Survivability 
Paeng . 

With this modification, weights are not independent 
of alternatives. Unfortunately, AHP cannot make this claim. 
We can now consider our units of measurement to be in terms of 
applique armor and a 105mm main gun. We will give this "unit" 
a label of SAPF105. 

b. First Approach 

First, of the original alternatives considered, the 
lowest, or least desirable alternative is given a rating of 
one in one of the attributes considered. That alternative 
becomes the standard to determine attribute values for all 


Other attributes, and we call that the "Standard attribute." 
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Attribute weightS are normalized in a way that this 
modification does not change (that is, the sum of the weights 
of the attributes will still be equal to one). In ottem 
attributes, ratings could be much lower than one. These 
ratings, or scores, will depend on the "Standard attribute" 
and will be scaled accordingly. An additional alternative 
could be even less preferred in an attribute than the 
Originally least preferred alternative, so it 1s possible to 
have a value of less than one in the attribute containing the 
"Standard attribute" once additional alternatives are 
considered. 
c. Second Approach 
The second approach rates the highest, or most 
preferable alternative in one of the attributes as one. Once 
additional attributes are considered, relative values of 
higher than one are possible in the "standard attribute.” 
This could occur if the new alternative had a more preferable 
rating in the "standard attribute" than the original 
formulation’s most preferable alternative in the "standard 
attribute." 
2. Tank Problem Solved by First Approach 
This section solves the tank problem introduced 
earlier to demonstrate rank reversal ina way that avoids rank 


reversals. 
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a. Original Formulation 

The original two tanks are evaluated in Figure 5 
using the approach that gives the tank with the least 
desirable characteristic in one attribute a score of one. In 
this case, survivability of Tank 11s the "Standard attribute" 
from which other values are derived. The decision maker has 
determined equivalency between survivability characteristic of 
applique armor and firepower of a 105mm main gun. The result 


is a different relative value for each tank, but still Tank 1 


ee rank 2. 












PT TREBUTES 
TANK RELATIVE 
SURVIVABILITY FIREPOWER VALUE 
(4/9) (5/9) 
TANK 1 ie 70 1.5 1.2778 : 
TANK 2 1.2222 
Figure 5. First approach with original formulation. 
The relative value scores are not meaningless. The 


relative value of a tank is in terms of survivability of Tank 
1 and firepower of Tank 2, because Tank 2 happens to have a 
firepower rating of 1.0. Its firepower attribute, a 105mm 
main gun is equivalent to the "standard attribute." A 
theoretical "worst" tank with Tank 1’s Survivability (applique 
armor) and Tank 2’s firepower (105mm gun) would receive a 
relative value of 1.0. Tank 1’s relative value of 1.28 means 
it is 1.28 times more desirable than this theoretical tank. 


Recall that we developed the label SAPF105 for the "units" of 


45 


measurement of tanks in terms of survivability attribute 
applique armor and firepower attribute of 105mm main gun. 
Tank 11s "worth" 1.28 SAPF105s, and Tank 2 is “wortn" =i 
SAPF105s. 
b. Additional Alternative Considered 

The original two tanks, and the third tankjijaue 
evaluated in Figure 6. Again, the least desirable 
characteristic in one attribute (survivability) receives a 
score of one. Once an additional alternative is considered, 
it is possible for an alternative to receive a relative value 
of less than one in Survivability. This could happen if the 
new alternative had attributes less desirable than the 
Omir ginal problem's least preferable alternative i 
Survivability Ciamik< “ae The result of considering eae 


additional alternative is the same relative value scores mae 


the Eirst wo tanks. 


ATTRIBUTES RELATIVE 


VALUE 
SURVIVABILITY FIREPOWER 
(4/9) (5/9) 


a a 





Figure 6. First approach with third tank. 
Tank 3 has the highest relative value. Therefore, 


the decision maker’s preference would be Tank 3 > Tank 1 > 
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Taimk 2). There waS no rank reversal between the first two 


eanksS . 


Unlike AHP, this modification 


(1) did not have rank reversal, 
(11) maintained a ratio scale of relative value. 





c. Duplication of Alternative 
A fourth tank 1s now added. Just as in the 
previous example, it has the same characteristics as Tank 2. 


These four tanks are evaluated in Figure 7. 


| ATTRIBUTES” 
VALUE 
SURVIVABILITY FIREPOWER 
(4/9) aa 
pes [1.2778 
TANK 2 1.2222 
TANK 3 1.3889 


, CLANK 4 a ta ie ee 2, | 


Figure 7. First = with arias tank. 






RELATIVE 








The first three tanks receive the same relative 
value scores as they did when considered previously. The 
preferences for these four tanks are Tank 3 > Tank 1 > Tank 2 
Smebank 4. There is no rank reversal among the original 


alternatives. 


Adding the fourth tank 


Cr) did not cause rank reversal, 
(jw mMartntained a ratio scale of relative value. 
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d. Analysis 


If we let R.,, be the pairwise comparison matrix 


2T 


obtained by pairwise comparisons of the relative values of the 


two tanks, we obtain the following matrix: 


ite Se, 
- _ Ti 1 2.05 
an 1 On SIS 





If we let R,, be the pairwise comparison matrix obtained by 


4T 


the pairwise comparisons of the relative values of the four 


tanks, we obtain the following matrix: 


G51 ee fies: I4 
yes! 1 oer Os oO ro) 72 alle 29) 'S 


ree ee acs 8 
$F EN OO mea eae “Se eee 
T4 10.96 1990 S8cneed 


The upper left 2x2 entries of the matrix are identical to the 


fenena io ccm Therefore, the ratio scale was maintained. Rank 


a7" 
reversal will not occur because the relative value of 
alternatives will not change. 
3. Tank Problem Solved by Second Approach 
This section solves the tank problem using the second 
approach described earlier. The highest or most preferred 
alternative in one attribute receives a score of one. In this 


case, we select Tank 2’s Survivability attribute as the 
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"Standard attribute" by which other alternatives within that 
attribute will be measured. Let us again assume that the 
decision maker, in order to "link" the two attributes, has 
determined indifference between Tank 1’s_ survivability 
(applique armor) and the firepower of Tank 2 (105mm main gun). 
Since this is the same equivalency established in the first 
method, we should expect to maintain the same ratio scale for 
this method, if it is to have merit. 
a. Original Formulation 

The original two tanks are evaluated in Figure 8 
using the second approach. The result is a different relative 
value score for each tank, but still Tank 1 > Tank 2. 
Essentially, what we have done is change "units." Now tanks 
are evaluated on the basis of Tank 2’s survivability (reactive 


armor) and Tank 1’s firepower (120mm main gun). We will label 


these "units" SRAF120. 
















| soarpripures sd RELATIVE 
TANK VALUE 
SURVIVABILITY FIREPOWER 
(4/9) (5/9) 
TANK 1 O67 WH) 1°.0 Ome ols 
TANK 2 0.8148 
Figure 8. Second approach with original formulation. 


The relative values obtained by this method have meaning. A 
theoretical "best" tank with the highest possible rating in 
each attribute (this would be a tank with reactive armor and 


a 120mm gun) would receive a score of one. Tank 1’s relative 
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value of 0.85 means it is 0.85 times as desirable as this 
theoretical tank. From units derived above, Tank 1 is worth 
O73 >7SoRAEIZ0Ss 
b. Additional Alternative Considered 

The original two tanks, and the third tank, are 
evaluated in Figure 9. Again, the most desirable 
characteristic in one attribute (survivability) receives a 
score of one. It 1S now possible for the new alternative 
(Tank 3) to receive a relative value score of greater than one 
in survivability. A rating of one is based on survivability 
of reactive armor. The result of considering the additional 
alternative is the same relative value scores for the first 


two tanks. 


| sarrRIBUTES) sid RELATIVE 


VALUE 
SURVIVAS DEI y ET REPOWER 
(4/9) (S73) 





Figure 9. Second approach with third tank. 
This method also avoided the rank reversal encountered in the 


original AHP formulation of this presten 


This modification 


(1) avoided the rank reversal of AHP, 
(11) maintained a ratio scale of relative value. 
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c. Duplication of Alternative 
- EQUGEM IEamGe wlenewtme Same characteristics as 


Tank 2 1s added. These four tanks are evaluated in Figure 10. 


Sie ATTRIBUTES 
(4/9) (Sy), 
PTANK 2 


Figure 10. Second approach with fourth tank. 









RELATIVE 
VALUE 








Unlike the example of using AHP with an additional alternative 
added having the same characteristics as an original 
alternative, this method avoids rank reversal. The reason for 
this is the relative value of an alternative will not change 
unless the decision maker makes a conscious decision to change 
weights of alternatives or the rating of an alternative 


meenanse Of reassessment or new information. 


Adding the fourth tank 


(1) did not cause rank reversal, 
(11) maintained a ratio scale of relative value. 





d. Analysis 


If we let R,, be the pairwise comparison matrix 


obtained by pairwise comparisons of the relative values of the 
two tanks in the original problem, we obtain the following 


tiaerix: 


sy 


Fab 92, 
1 1.05 
Go 6 eel 


ee 
2T ee 





If we let R,, be the pairwise comparison matrix obtained by 


pairwise comparisons of the relative values of the four tanks 


using the second approach, we obtain the following matrix: 


il, Spee T3 T4 
al 1 1705? 0.22 eae 


= T2 | 0796. 1 eo ice 
<P 731) 1.09 1.14 See 
T4|0296 "a "Oe e) eal 


The upper left 2x2 matrix is identical to the 2x2 Macias 


obtained in the original formulation from R.,. This Stew 


that this approach maintains a ratio scale when additional 
alternatives are considered. 
4. Conclusion 


Both approaches maintain a ratio scale of alternatives 


and avoid rank reversals. Because R a= R on: either approach 


solves the rank reversal problems identified previously when 
using AHP. Whether the decision maker decides to rank the 
highest alternatives in one attribute as one, or the lowest as 
one, he or she will obtain identical answers. That 1S, @eme 
hierarchy of preferences will be identical and the ratios of 


comparisons of alternatives will be identical. 
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V. CONCLUSION 


There are at least three modifications which should be 


made to AHP before it 1s used in a decision making problem. 


A. PAIRWISE COMPARISON SCALING 
The nine point scale used in AHP is overly restrictive. 
There are at least two ways to modify scaling procedures that 
eliminate a significant potential for inconsistency. 
1. Decision Maker Defined Scale 
Instead of using the value of nine to signify extreme 
importance of one attribute over another (or one element of an 
alternative over another), the decision maker should define 
the nine point scale based on the characteristics of the 
problem. We suggest defining the most preferred alternative 
aS nine, and the least preferred alternative as one. ial 
intermediate alternatives must fall within the scale. 
Similarly, when making pairwise comparisons of alternatives by 
attribute, the extreme alternatives should be used to define 
the scale. 
2. Open Ended Scale 
The decision maker could alternatively use an open 
ended scale for pairwise comparisons. If the decision maker 


feels there are several orders of magnitude of difference 
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between alternatives, he or she could use a multiplicative 


open ended scale to determine preferences. 


B. WEIGHT FITTING 

There appears to be no advantage to using AHP’s eigenvalue 
method of weight fitting, other than its ease of evaluationme 
the AHP consistency index. But since the consistency index 
standard of 0.9 is itself arbitrary, evaluating one’s decision 
making by the consistency index is, in reality, not much of an 
advantage. The least square error (LSE) method of weight 
fitting, not surprisingly, produces a  smaltem Sennen Of 


course, LSE was used to determine the error of both methods. 


C. RANK REVERSAL 

We have demonstrated two ways to avoid rank reversals. 
Both methods depend on the decision maker to establish and 
maintain a system of "units" in which relative values of 
alternatives will be measured. 

1. First Method 

The decision maker selects the least preferred 

alternative in one of the attributes considered. This least 
preferred alternative, which we call the "Standard attribute" 
is assigned a value of one. All other alternatives within 
that attribute are assigned values based on each alternative’s 
comparison to the least preferred alternative. The decision 


maker must also relate the value of the "Standard attribute" 
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to other attributes. This is an essential step to maintaining 
a ratio scale. The decision maker will decide a ratio between 
the "Standard attribute" and an element within each of the 
other attributes. No matter how the problem is changed from 
this point, that ratio must remain the same. The alternatives 
must not be renormalized with the addition or deletion of 
alternatives. 
2. Second Method 

The decision maker selects the most preferred 
alternative in one of the attributes considered. This most 
preferred alternative, which is the "Standard attribute" is 
assigned a value of one. All other alternatives within that 
attribute are assigned values based on each alternative’s 
comparison to the most preferred alternative. The decision 
Maker conducts identical steps as performed in the first 
method to relate values to other attributes. 

3. Results 

Both methods produced identical pairwise comparison 
matrices at every step of the tank example problem. Neither 
method will generate rank reversals when the decision maker is 
consistent. We demonstrated use of invented units as labels 
for the magnitude of the relative value in each example of the 
use of each method. Since the notional Ree were 
consistently used to measure the relative value of 


alternatives as additional alternatives were considered, the 


ahs 


relative value of alternatives never changed. Both methods 


avoid rank reversals and Maintalm a vae occas 


ai 


APPENDIX A 


AHP AND CHOOSING A HOUSE 


This example is taken from T. L. Saaty, “How to make a decision: 

The Analytic Hierarchy Process”, European Journal of Operations Re- 
search, Vol. 48, pages 9 - 26, 1990. The purpose 1s to compare three 
houses, A, B, and C, uSing eight attributes to find the most pre- 
ferred house. 


SATISFACTION 
WITH HOUSE 


NEIGHB. GENERA INANCING 
CONDIT. 


& 


XE SY 


HOUSE A HOUSE B HOUSE C 





The following matrix shows a pairwise comparison of all eight (8) 
attributes, together with the vector of weights produced by AHP 
get add to 1.0. 
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PRIGR i 
VECTOR 
es 


A Matrix and Priority Vector for House Ranking Proeoimem 





The following eight (8) matrices show the results of pairwise com- 
parisons of the three houses on each attribute. 


SIlZE 
A ls S PRIORITY VECTOR 
A ak 6 8 Poe 
i 5 oye 4 elles 
ae ya BOSS 
By =o 36 
YARD SPACE 
A B © PRIORIMY (Ee ner 
1 5 4 .674 
ome iS 2 laa 
Lia a3 il i226 
Bone ois 
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TRANS PORTATION 


A B c Ele a vee rOR 
A i 7 LS: $250 
B I (a Wis -Ue4 
c =, 8 i fa 
By=3.247 


PeeeRN FACILITIES 


A B e FREORI TY VECTOR 
A 1 8 6 ey 
B ims 1 Wes) 060 
Wo 5 Ik ILS 
a= 37.197 
NEIGHBORHOOD 
A B é EelOnil yey sCreR 
A i 8 6 Pea 
B io. 1 1/4 Sis s 
6 4 1 eel 
mv=5.136 
GENERAL CONDITION 
A B e PelOni ty VECTOR 
A ik 2 pelle 2200 
2 A ul .400 
2 Th 1 .400 
v= 3.000 
AGE OF HOUSE 
A B C Ron ly SJEGTOR 
1 Ih 1 Me ae 
Ik Ih al Gee) 
al i. il yo doy 
ae 3 . 000 
FINANCING 
A B © PRvOnR rity VECTOR 
Ih ee ley es) me 2 
q i 3 260) 
5 epee al Pate 
v=o. 065 
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LOCAL AND GLOBAL PRIORITIES 


i Z 6 4 3 6 7 8 
(.173) (.054) (.188) (018) (.031)( UsG) te aes 


A 754 .233 .754 .333 .674 ./47 (200 eee 
181 .055 .065 .333 :10) (060 24003 os 
.065 .713 .181 .333 .226 =193 240032 


From these AHP shows that House A is preferred to B 1s preferred 
One 


APPENDIX B 
What follows is a demonstration of how to determine the 
least squareS approximation of determining weights of 
attributes. This demonstration was performed on a 3x3 matrix, 
but the reader should be able to recognize the pattern that 
develops. 
Recall that if the pairwise comparisons are perfectly 


consistent, the following relationship holds. 


a ae Ij; given in pairwise comparisons. 





The error for the comparisons 


By definition, main diagonal entries must be one, or 
f7e-1 Vi. Expanding the error term for a 3x3 matrix 
a Ras Z es 2 = 2 
@ = (W,-@27,2)*+ (W,-,7,,)°+ (W2,-@,72,)°+ 


(W, —@525,)°+ (@;-@,73,)° + (@3-W274))° 
The reason the above expression contains only six terms is 
because the entries for 1=)j are all zero. 
The row entries of the solution matrix are the respective 


Partial derivatives of the error with respect to w,, where 1 


designates the row number, and setting the equation equal to 


zero. For the first row, 


Ga: 





oe ee 2 = = 
2W, = Wil5, Wil3, wey | ee ba he ae T,;3) 


_ a 2 _ 2 _ _ 
3M, = 0,7 O)75)° —@y lq, O53 (245 = 254) 1 On eee 


The last step, adding w, to each side, was done in light o@ 


the solution we are seeking, Ew = nw, where E is the matrix of 
errors, w is the vector of weights and n is the dimension of 
this vector. Similarly, the partial derivatives with respect 


to w, and w, can be determined, which lead to the follGwaae 


matrix representing the solution form of matrix E. 


_ 2 _ 2 
OR args (fee (Tee 
_ 2 _ 2 
(2557353) a a ee) (a 
= ae 2 
eerie (F5,+73>) (ae Eoaae 


To determine the error when using the dimension method, we 


solve the equation 
RW =N®. 


Recall that in the case of inconsistency, there is no exact 
solution to this equation. There are many ways to approximate 
the vector w. The procedure for deriving the dimension 
method’s approximation, obtained by multiplying both sides by 


an identity matrix of sizen 


Gz 


Expanding, 
n 0 W, Zi, 14> Tin Wy 
Ww ié W 
2 21 2 
= = OQ, 
0 I GW Di Jan GW 


This iS equivalent to: 


yey te 2 
nN 
Ol roe =O 


This leads to a final expression for the error matrix for the 


dimension method of estimating weights shown below 


Wy W, 
W, W O14 Cin 
R - i 
| W, W » | 
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APPENDIX C 


SI ZEN COMPARE WostZe 
METHOD ERROR WEIGHTS 
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METHOD ERROR WEIGHTS 

SAATY > (735.902 W230i 
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METHOD ERROR WEIGHTS 

SAAT Y=»: 17.820) 754. 065-151 
DIMENSION = 231427 07 2202 
LEAST SQRS : 7.477 .780 .084 .136 
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METHOD ERROR WEIGHTS 
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LEAST SORS.) 300G 535233307242 
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METHOD ERROR WEIGHIS 

SAATY  : 4.439 .674 .101 .226 
DIMENSION : 3.627 .655 .105 .240 
LEAS LSORS = 3.026 2697 .i lee iss 


MODFACM COMPARE WSMODFAC 
METHOD ERROR WEIGHTS 

SAATY =: 27.540 .747 .060 .193 
DIMENSION : 17.584 .707 .067 .226 
LEAST SQRS: 14.946 .784 .078 .138 


GENCONM COMPARE WSGENCON 
METHOD ERROR WEIGHTS 
SAATY  : .000 .200 .400 .400 
DIMENSION : .000 .200 .400 .400 
LEAST SQRS: .000 .200 .400 .400 
FINANCEM COMPARE WSFINANCE 
METHOD ERROR WEIGHTS 
SAATY : 5.860 .072 .650 .278 
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Meer SORS : 5.438 .087 .670 


HOUSEATTM 
00 0 5.000 Se 00) 0 
e200 ee) iO) OREer ais, ae e)0 
we 5 3 oe VOU. 1.000 62000 
ve 143 0220.0 Orlow 1.000 
Celso / Cee po Oe 5.5 5 slo ae 
eo / UeeooS O12 yl) east 18) 618) 
B00 moe 018 OL O'/ ING) 
meow § 7.000 2), OHO RE 8.000 


"on 


Oy (Ul So ca, SP eee 2 CI OO 


HOUSEATTM COMPARE WSHOUSEATT 
METHOD PREOR WEIGHTS 


BSeANyY:259 173 SOD liste! 
MeNSLfON:858 -.031 .081 LS e: 
Moame2o! 2/9 .447 nOoS 


.000 
.000 
= ORONO 
2o00 
SCS. 
POU 
000 
.000 


Wile) 
m2 
2027 


65 


20.00 
On) 
.000 
ae, 
2 SL OK6 
OO) 
20:00 
000 


OV Ul -- ee 66 CO) 


saat 
pas 
041 


Pops 
8 
.000 
.143 
0 
8) 
2000 
000 


bo Es (eee) OV eee 


056 
a2 
041 


ee eC PE a a a 


milo 
mly0 
eZ 


Ps 
.143 
50 
ee 
lio’ 
oy 
.500 
ue Oe) 


soe 
.404 
soe 


LIST OF REFERENCES 


Saaty, Thomas L., How to make a decision: The Analytic 
Hierarchy Process, European Journal of Operdtromem 
Research, Vol. 48, (1990) 9-26. 


Schoner, Bertram and Wedley, William C., Ambiguous 
Criteria Weights in AHP: Consequences and Solutions, 
Decision Sciences, Vol. 20, (1989) 462-475. 


Howard, Ronald A., Heathens, Heretics and Cults: The 
Religious Spectrum of Decision Aiding, Interfaces, Vol. 
225--NO. 6, ~lSS7Z) iS 77 


Holder, R. D., Some Comments on the Analytic Hierarchy 
Process, Journal of Operational Research Society, Vol. 4% 
NGOwoia, y 41990), 20734884 


Roper-Lowe, G. C. and Sharp, J. A., The Analytic HierarGny 
Process and its Application to Information Technology 
Decision, Journal of Operational Research Society, Vol. 
A Rs GSES) eee Ci 2 Sv Sie 


Belton, V. and Gear, T., On a shortcoming of Saaty’s 
Method of Analytic Hierarchies, Omega, Vol. 11, (jeg 
223725 0. 


Hihn, Jairus M., and Johnson, Charles R., Evaluates 
Techniques for Paired Ratio-Comparison Matrices in a 
Hierarchical Decision Model, Measurement in Economics, 
Physica-Verlag Heidelberg, (1988) 269-288. 


Saaty, Thomas L. and Vargas, L. G., Experiments on Rank 
Preservation and Reversal in Relative Measurement, 
Mathematical Computational Modeling, Vol. 17, (1993) 13- 
18. 


66 


INITIAL DISTRIBUTION LIST 


Nox 


Defense Technical Information Center 
Cameron Station 
Alexandria VA 22304-6145 


mmeGary, Code 052 
Naval Postgraduate School 
Monterey CA 93943-5002 


Professor Kneale T. Marshall, Code OR/Mt 
Department of Operations Research 

Naval Postgraduate School 

Pemiberey, CA 93743-5000 


Professor William G. Kemple, Code OR/Ke 
Department of Operations Research 

Naval Postgraduate School 

Mompzerey, CA 93943-5000 


Professor David R. Whipple, Jr., Code AS/Wp 
Department of Administrative Sciences 

Naval Postgraduate School 

Monterey, CA 93943-5000 


Professor Thomas C. Bruneau, Code NS/Bn 
Department of National Security Affairs 
Naval Postgraduate School 
Meme ne Vem 9 3943-5100 


Broressoun Paul H. Moose, Code CC 

Department of Command, Control and Communication 
Naval Postgraduate School 

Monterey, CA 93943-5000 


Dr. Michael R. Anderson 

Operations Research Analyst 

Combined Arms Analysis Directorate, TRAC-OAC 
Heine beavenworth, KS 66027 


CPT William H. McQuail 


6432 Woodville Dr. 
Falls Church, VA 22044-1429 


Cw 


Copies 











DUDLEY KNOX LIBRARY 
NAVAL POSTGRADUATE SCHOOL 
MONTEREY CA 93943-5101 





Set erat Mabe baci td = mt 
S9- stealf. tf DUDLEY KNOX LIBRARY : ' Bases Ts 


= eh, 


“se b4 
2 LOY as 
*s. fa 4 ek 
' 5 . % ( 3 ; ‘ . 
ae fire : Ale 
; - te , | : : ' : oe ; 
_ an s . i ry > . . 
fag 5 , - P 
ky PS ed : 
Chess e pt ‘ - . . «8 
ort ae 26 : ¢ e oc 
oa ° PSH | ° s 


as i eh 3 2768 00307990 6 pee heads S Dle a ee 

















veg shee: 
it sm 
LPR 

















ae 










oor 


pags 
pes } 
























or 
oo baie 28 
A opr Tapes 


tx Siege ee Bed 
na a Ped ee 
der pe 

va o 

ates Mi, OO oblet dP, 
e Yighstapedel sat 













a ifs 4 “ahs meng 
SoS ade! ie 


oh rOre oa ™ 
eae te ee 
ove ened 


te 


ene ames 


Upbebecas saeed 
panecrdras Betas B kas 
wh te paps wo aban y 
is ne 2: ne Hus ue) 


~ 





Cay 
























































































































































































































































































































































































































































° 
Hered ee ' i 
‘i, 4, ete 
Piet UA & ote * 5 a Ke ve Pe te 
ne Big ah thipdve tes 2. Oe EL ‘ ° 
of ke hel te Gia ts bee ecg ant ps ay ihn’ 'e RA a a . 
e of @ sh «hee, . ee os s 
io Bar tos rae a seine et ee fat @ 3 
brs heh aeaai ¢ Fieger. opts i des (4: e 
ob ¢ & _oa 
a “6 ve 5 Pade agts - * 
5 te of 1 hed; ;* fen Le % ‘ e 
pases ae as . 
oe 3 ions spate oat 
. [I at 4s Ow 
ry bay Fe Pio at Le, idiots ri i Frog lh & sgt regen 
e . al - 4 
ees ee SPoagiekece ie maeatts Ge rantahs Fe a ee af > 
pe oe A epeeis vA teas. 43 ghee hele oy A or ptt vt ESTE f Gi att * yAi 7 
Aad estsdaiyie Je Pat ity nig #2 o's heats at o ist! Bivens EN cy G aie eee Bae 9 A 
tage sies « A fee Settat i bens toh wou ei s is} het (ar ea 
soon a as aegilte hist: a) Pen tr. | ia ° 4 
fare, Yast ny! Pa beb eis tentd se: ; A ; 
Satter tats SSN et eT la -  E “ Se 
t ase « ge . * * e 
ery etal tas oh co 3m ieesemercas, (6 18) a ? ane F * . 
Bs atta Pe iat seas) se owe * 2 © . . - wie - 
etl eh hanety! Ae eid Fe mir 2% ete +6” . Pry r . ‘ 
aes re Fase pa aory hey ot “ Tae ce Sip p a 
eae rote ‘gigs Pon (ren) 5 Ai ta ee ea "et . « 4 e 
Pee PL} files ed tik, 4° eo ener t] Sieh ener ge ¢ Cm ‘ . .” 
ot Op Saeed fe 9 2108 ofel o Rae, Wa of »f = ,tf etet te ’ er - . * 
of Fedo “et wie sel 3 ‘ +4 ; -° eee Sa neo sg os nase 
“at ait ee gd a YPAY. RA dake yy A aplatites Uc apreeME be race + 8 Ape 4 .- 5 F , . 
Lo aay Peet caeaes ose he or, tae 3 ' ‘ WP Ne? mate poe > ” 53 a a) . d os a) 
BLA ftet sake uf be, sigd t >> ‘ » seaedn ee cee Se . ’ ye ee 
ye oeratt ih rts fa Repeh al "fe v ay "S45 Pu. FT prvrrhic pase ’ Sia? @ she ges ° , ° , ee : 
a atts oF @ © adage? : i ot? ia Begs @ . ae lee se ’ . -* *, . . Ld 
ote eae ete ae Pe ee: tates Spe abt ate Ra on ae Opn halk 8h thet po wT Sy ea OR i oo wee 2 “eat ee ’ 
Sere egg te piepnce. rae read Be, eret ae. a oo 1, Pa see rere a + Pa ie ee ae dd av #% @ i a eee ete Oe & fom % os Ad a 
Sake doe ae. hse, ? Pate seTpeahe® o aah 4 Age 8 at yethss tet te ee 
a of mine eer iit ibe cebs ait oe eel dy he starts ’ ta «byt « . 
aye rrp . Mot ee ds)t0 eyes pate saitiead | Cro art ett, 3 24 2 6%, te . ° 
piehed Pt Aabetatete i ; rr if ae? | “Beaty ak ne! dies fis.” int o § «4 ‘ a 
1.70 at of Bene haFvEAl asi Bal) ay ule 18:0) ae shia *epigeenice nest i) . te we 
~~ ae oh ws C Je . ers ve lamas @, fps, CH viet TaAee pats 7 
oy, ars r der ky ie get Sepr> a girs > * =* . AS 
! eee 8 tat gh ny i Ager! S> oe ‘ eel "6 o6 i 6 ‘ 
*al, ES! 70d 3 ehity a! ? Reel” wae i! ' aa? et a a a ‘ 
rus ee ode) ey yeahs vi 3! ne she s in UL Fanpenss ia fisggy BR Pigeass & | F; ' 6 oe e 
re te sees Sit hae Paes er Sy rd oY shen o> Pier hs aa ? es 8 “8 wees ae 
ot or abtel tag aie ie nae Re . 3 sect Ap “ngh Para iG . oe *« » © &e 
Bab iae. 8s: sy Food Pale Sf La a ee 5k ye gy anne ab Belg ut ee a Site ‘ Re ate a6 Wt lets. od) ry eod.e6 ao 6 * 
TAmt atass See he hal ol rae Cae ie pe eeliee oes Tee | 1S nah Pow ” Perens Ser Se | ve ee ‘ c ® RZ 
EA 8tphe ohn hy Hie af Pr aa oe 1 rR ahs ey! cr sehawe’ ayt Tv. Po ¢ "ek haeye + 1 8 mw % . wa aes 
tae ioe Siete? abot bral ed Bad gies dyi at Sa Re nee Paks! opr ys oe ik & ie acest , MF bettee Feretigs: *: eA 6 syde 6 0! > Pe 
wath yt 38 iietere fi Ebest gad od et sinks e ihe pt ; Bateau ab gfe! 7 ee areranate ° * 
Pee a ae re iis Be hel "Pgs ate °3 ‘ e ce a 
we pecag tote Pa thel Ww nas 1A ‘aby &A; Chee a He boas Pry . Ria. * ot re: . «® e e ote 
od at a? ll od rw Pinity gis cok 0 yee hada. Pe ged eakl rADvw'sl os ah 4 " b.2iait = Tiga e a * 
Fide» Pap aryl ot Dr, 6 pit whys orever! gates oan sb wemytaa: ys ar fy - ‘ 
iat bar oe pretest Erg fab eateig baie bed Faentis ons bee be rrae a0 808 4 Ad es @ees . . 
"5 aoa oF of Regie Lad pated. aNd eR Ar re Wear pest é re here ~ ” 66 s¢ 
* Fe ‘en fst bie dacye: Leb Tete oe Oe Pld) fe beg ibe hat gre « : a) "6 . ra » 
ae OF ae $i otk -. bys * > ws M4 78 +f tod © O90 L 
at A re oe end pe) a Ag pee? m2 te j oS fee tte ry ae ‘ * ¢ .? 
ware? ad re Posed, “a PAT OF gadet, ok ay) Lee ost ae a ep © * 
tee ed Shas poe vibes 6 Te afte ol byt on te = $ aeee . 2 
28 oat Ss it ght #258, # heheh copie « cg tts Meer Le aye hep a ‘Reus : ‘ ° “¢ = oe 
caf A dant fd glarag bit nye oe ag * ; Hehtl 9. ve 1P aut ee peehgi me 1%; fon ae i . on ‘ * 
I”, Fae bia ki PPeea) Ae 4 ~ ina of th, eee Fi otto "6 ory 4 . ar) r * * * 
‘ POY . « went gl 5 . 
FEM gz) %s a, ae , oe tee: ee Be ee <i al * 
a yer tat | are é a . | ES pas. re * s¢ ~ 
5 a ‘nF perg heya.” Mi s fr) ong . ’ 
$s etsy ead ne ia is ace eS oe a. 2 . . 7 s =o 8 
‘ é A ' WI te . . . * 
ea tie , caked Ha tt es ey ¢; . oii net we 3 ” ° shape Ni ge Pye stan esths Ye a 2 * 
i set a: reser tes Segheaat coal Sy ae gatas cory wet pies woe tae, : Fyashia: iaek. edu festeas en Sd a rR ade le aes 
i fy a rt BY telet.m, noe tes 4 te y i oO 5s, og optee a . ' 
- iis Job Sn gh 2) iho eS eae AHP G ANE LT BRA: *hes 





OY Rh ae ye Zh ae! 

pa orl i ve te Lee a et ear a et i. pA SM 
3,407 yrs sat ch ah Le 

tis eat i Palese ns at of Rey ieie lade thd 

bg ath] Ree, iA 

au Ae ae eae 

' soe Pa = - oe ot BS 


hs Baas . 
% 





Ooi wt 8 Bae bh 
Shai! ete 7 af b 
eFooaDger® * a) 
op: oo than (esa a acre 
r,t “hb psy Me ot 

' ch ane 






















t seat) at :"} 
a8 fis 


me) & mente 
Pd ae 





rh 
ih Eth 
Poe a peierS 


BAS bet 
sf a> Fa.e : “ 
au 














o 


thee Ct re eT 
=8¢0@-s § gab Ceo 
= te 












stte sreeesi AUartare 04 34 the g0028, 81 Pee 

Fag 0048 ry oF . Pea 0% 
ree. 3. Ppenpre 6 
1 se efe goste bh 



















Mabe Meh iad h 
Oumsfak te O42 ad 












Ly 
ate He” 




































































Pagel Ad Li shat t ts 4 ¢ 
: Pa Pt ‘A Dat Ete at i Bes iib Maslosmecnetsne hee mee Ps} 278 athea oP Bec) 
95258 * ¢ Fat eg’ eee? ape “petadars tyes de 4 FF a om Lp 1e oe 
aM Parade Sifatgondes Rie OP 05% Pr, ’ Me s ipa + be paa’” e 
ae 0 an dt tite aP ft *s Pare * ra Sra 099) 80 0) PATS ae “hy a1 4} “7 i = s004 PSs / 
7 bl s « - 
ote alte att . ve 5 oo iN Hedy ie of rigs Spacers PRP ir ar $4050 00 ehehete a 


72%, are ert to yn ¢ .* araee tt Moony ‘ o4 


£89; 4 bd 
a ee ee ats g 5800 tes 


: te 
HF sea? 5h, aeN Bad 


ets! 





ey “tw. 
































































































































































































? H Ae, gh 
;,t eye, 4 a 4 of Oe eet 4 te 
“8 peters oR as en, Ropzer d- Wopte fF ape eet ove eae “eect Steg 48 
Mey Art ope nS ee Coe ba) Rss a.p phSetg gree de Fiat ws, +'s "te tea @ of 
vi ? Vie 4eri nr San ecne are ergtat Tertten "v4e,5% ¥ +g! goetar ‘2 
G tode ceiPe's eVere Aire hes SeNber GI fePee® “ * 
Dezgt it "> » tabs + $e a ste =o$Sdaet +313) ees i Gene a 3 a 
WA F08y Cabo Palodg tT) ‘ 3 Poe" on" oe tee 7 Om ad ig ony i a " “~ « «se ? +s Z es 
j a sti aaa apy rf NU heat ot i rt ers ae ‘ ea . 
eye Fumetpe oe aig ese % ae taba” Wyee’ on & a fa5% oF > aes ar. ae “e a : 
Fa2 See eat oy? DLN eo ohh» da pare oe oe tow ra ° erg ee stages * : $ « 4 = re ‘ 
a op ata ear es it 4 pata de 28 t yr el ee ae ee ee ee Barats ate . ne oo? 
xt fae "Fo oy eh Rw ae y trate’ Bee arte oO ch! see get ey é ; 9 te . ee te ante 
Ped a at Pe END? 5 Stee a. af de iit 5 oF kapery 7h See a Care (St Oe * foe re aie, eet ty eo Dee * bg tony grered ‘ ube. 5 ssee | * 
SEN. +r ets Vier Serytads tele oF (eReAdes ae 4 egshe rgsidem.b Sage ry atg 6-0 oe ae gene . "te 4 whe 8 ‘ « ot i" 
47.3 LO se or ates ter eat OMS Hm Pate eft’ 2 etary) da ae i in “tris ne ators & & ob DAS RD ire 98a cee 8g "yes ; - 
a ‘oi © ie Hara 4 gy : ; Pratt rea es he Roady get Eseries Mr ot ober r Aiy? dyin at te ade s . 
chs iF, Ss (ni Ve hye € ; ; aware i] se strutaeclai ne Ra ee Poke ia ate Fees eee : 
4 Be ayes ge) te éeF I, ry Rate ty i % ee hy pedhan oly abn ree Red, F3bev Le ' * Gel sea a8? om Bit 5 * 
pi beef yss: if atusiyirds eal: sbekeiyen ) erase Ne eee, Spates; Dies aed bus segedwcssterg, f eiaactep Ber ate Seta? | eo 
v4 Lele aera ob es cide Hate. os ckis eves nF * is aa eee Fur wee! aeye 18 ge a" fgtsegedes Ps . 
v2 52 ea Panitret hats He > Hey Laas sparpesees *e r 360 F. e ‘ 
yr pits? vibes 42 $2 areSee se. ae yeseeke ivbye of, Bs < . 2 Sapee 
i= i th 94 4 “© + ov a* . 
i hed Pewtig ce + : i labete a re cr le egne 4 g's o rr les et CeO e* j« afee* ° ad 
és oe FO weed vos we inmost F dread yeoettyae Serengnrs sete Cay! tage a 0 step gt fe t"a-p on * « ‘ 
Le sees Nit aa oust api Late ye ae Wags agemesan gy. os Weoe ett te "ww 6 + Dect %a s a | | ae # ® * ‘we ; 
ag aha ase een Lie : 





e 5 e ’ 

Sok att ake sea 
* Teli at tal 

Botta Ears Ty Pb ve sty jet 

na “SLRs, ait ripe ated} 

eo "y cei oy he hatte wy 

fi 7) 





. ae a 
“PS aeayh. - £AS 

Mat ep ath Here 
S9aere OF Pat Sm BLE og A Berkey e Fh. Beers? 
iii oF Ais: ” Na ye ‘ Ree oie 





' ost gh 
te mdse 4 











an He Paes is 






































































































































































































































& meen + Pred event saeny ston. = Gabi eth pore Per & 
id im erie ree vate + ieee aS pay i & fae Reieerie e323 Seis Bess Bites Shh te fe 
ea ee oify aH ‘ as teed Pepe fas ~ Fys ‘ Le tF ~ ef setF eb hey? 6h ote FS 
2 et at Mr ats > i Pa Pee a > ee ob, 0-4 Garte Coa TY a . si iwieees Aaa ete en eee d 
oe hy phgameets he ms 8198 they OYre @ oF MeO, pene ba Py hss, th Fuferlr oe ® 5 So wil 
tas aMrtcaNied 1F 7 Shi Re to af + bats Ad sap ote a) A * a Pr, FL rs 8 he 3 | sy Hi * e ; 
: ® Les aa * . e tie abe en? set ‘ $ ? “ wae A eo 
wo} Pars ielsto ? A, p a F + SAG woah Vs ate aaa Tay, ” 1 et a Date.” hs Pret fe PS “pais ® weg + eatery pitas “F +" teap,e Bprsote rr ? ¥f woagnete eehbe OH APG ender coe, # ecdagrs & - a Le ale aaa ~ yee 
= erin ot 2 Sie Bayes bedi ey ay SB as hs i? . tiers art u AyFedoup’ | a DOat hs ret if Sanat Reset: 3° yy 4 Ps 3 hs gine: 2 ot $222 2 § 803% @ et pegs Oep & 8, ote a uote * 28 » atis * 4 * 
. Pareres, 2 Bh Pee, racy as nen *: ‘ vat, i iy ES BRET apm MeFi PRR! ? r -. rer ehe Tabeit Re gtitend  sxlaom ch? : “ a a § s28 =o CMEEDERIa 06" Papas aeeyy sets : pL ah Ye iyi eae = . 
tras ng bee rt ns we $3 " 4 ; hat, tor ‘ Dy! PTY a setintan totrms MIN DE, 3, geass sos, ta : Ao Ie Ue oS aol toee i 4 eee 
ye ava 4 . ae t i eS Ge 200, Penk = ee * ° > 
SNES B SS shia! tp se? ® i att falas os ghaat Sia A a9 i ; $ qn FR, 5erws we “et ¥ tohg e vo } nears _ Cee e tate . Se on « 1 e * * « 
Pola e sein MP4) Fe 600853 = a) a } iy rails ery natd. das oes Tee Be be yest tig 2 tau Pe atrtud ove, ¢ | a te? cfm * 
a Sak RRP HEISE ST ou ene sit : acter or Weare ta ap een eee Dela t 's 
2 ) i Pry aa? . i ; * dem = tat g% ry 4 oe 
a Bae se ices 43 reed a ty ES ate 4% pi Tee 8 ae ‘shi: aii Pas LA ou aae Fu tua a ti Ay patina s/aver > Sei ie lecce as 
ied a certs: tegen ay iy ee: iss Puen : eT Nahe Ses sietst tim nnene be) toy esa 
‘ ie a4 a ste bee a te rereny st 4% feossgies rfe8ed Moubestipae’ hot we ° % 
ie MASS ary Hb epPS ee ah ati i sh +t? hte poriners Pt) ae by 3h @ Ha -U3 te 1yoe Shed T,2us meres © Oa" phe Aue? ¢ 
iain +25 alice? ity one 6b Ae ota qe AP Sbeiey Ver, apn? Oo yh seat taa ithe Hee Bor yeas vat PabyeY G MBe iy Hepes S Mae Sts shee Ehender sev 
iter eas 335 $ e, tS wo caityhy a poh SAviccenfur it F581 tye De ECL) em pad nips 4P ches ae I. 3 Pd : $808 o Mee Seus 
Ee . M4 mn, spre en ty: © SF ete wage 3 
4a? So fol fade ? ee maa he: F } oat ’ abet 8; crite oye Jy. Sar See PL) os » 380) HuPene, 
! t + so pt Ae 
< rite * aya ete t ate wehbe try «ep! Seehah heci ? 
eon ff aide F (H 9% "dy }! oe Patt) a r ts t 
y, sce ; Kee. r ie Oe Fe Jy 2 Shep tpacen oF i 932) e* tum ze %e és 
a BF ratbe ladeay pbs 2? aie? ae te? 4 4 bee mH & om yes © 
beri mn? + = fiet ye? depha Jeeta fa eM Ff rere 9 rs 


fe silva 


, ia: Ge 
at} Ainge, MP EAS eet oT NSEAE BL OPADe HORUS TE? pate 1H vst t etme ae ysacte's 
' 

"> 


“ae +7290 S450 > ee pei mete » mS Ge CS . ajbias 1 0&ea* 
ae tewWide - Ee we oe he owe ue 





= 

sores" es z 
OTE R Say ubtvgdily 
ae % ahh’) Be 




























re eS oye tnoeny OP *,5 F dsdsen AM Pie! it) erete t oF 
rt isi Peta LP % A Hed ; sf 7 °. oh aeeh La wath = 8"aF pues 
b ate hs eae te ba tee Pirg 0%, ose C ion dope tr 
b. arse Bat be freee Fa7y ne) SP ann "457 cf > Fianys ees ie: Ai ete Py Pe tole shires s s2et les 2 ary fake t 4 owt Poe 
aber. xt bah ag bo aia) enmiasts aye St Be pitty yee Sear v3" 




















Oye 
aie witb t 
wt a bty, s 
Vote, Fan typ? ye 
eT ty 







ws Weve t=" eBVARS | as Reed ex AF aks 
Babe sae ela a Mica sieht 

‘ane t poe af Be ray 2h 
wAsee ee sy RY et ors SAiryt My, Pipes range ah 


oy 
ae yates ete be 88 tafdans 
Chae cea Weoogtt pees 





eeu hoe @ AL 
pee *s tae ob a Be 








































ST hes by Se ae otte Aya ws. Peder otn Pay? D9r yeh Ud 0 eee es 
a atin + Orch SP Se ye Hash pier a eat: sti) bei uag ae: wtag 6 
rH Ar {i ete hd i meena ide Y ° fone ain 1M eb Yen, tenesea 8 F ; 
Lt cecRms or iin RE nes hey HH arty} nish} as An Ee BY, .snfew yp rere & on tte g anes 





abet wy fede to S86h. hg SH” Se 
A Sey wf @ wage an . 
ance vache sele « eee 08 
ety oF a aptvoksers 8 & 
Bae 





% 
By fare eee ia zm 
erndied eh yeas ae ae care foek Ass, ey 
atrers Sivalite TS th Bdge g U8 g ad Sel Ay Sin ata tee gee aye Fh “oe ’ 
IE ses 4 Soe sopmoe & at hey SOY apd 2 gua’ Bpee a fe apfedess #49 948 
ot | eet ine g tute WEES! ZY sien 8 g-$ls dng see 7% 
FoF be uy ae vee dateges? As eat ix ui » Regen? aay Set tf, 
jr ey ey ye 


‘A 
ert ARIES Pa wity 
- ERE HS Tey’ seg Rave a Sa 
Regen) oy @, meet sf Sig sais Meade’ 
¥ a 


Tebe f nl 19 weap Ae aD ef Pale 
rs) i cag FG Bt x Me, if semi te ie sole Poe! ae 
i 

wes 


a 
Ls 
Pipead fer ats te 
jh ip BEETS be, 
Fe hile i tarde Sa cat 
ar AFPne #9) 0 US eee 
Pobyty me? aA ea sl La 
vse 


































‘ 
valk 
* avian ot “ty 
ep 






’ 
ei? 












































































































































































































































































































































































1h 
4 gene Oat wes See eg! reve ghee NS ot M iN tiaha: ot cat TCL an Bo taND apt “4tet ect ted te 
+ s ‘ hing ayn py at a r ac’ Corre ieee he by ” 
Sa Paee tyne bad < fs ed re s4 Mat WV} ay Serr ey tee bai eg x Saterday welts , t ‘ bis oy be nese Lo eines « a. | a A ee | hey SeVee obs x 
Om SA eye An ‘styaae wityte gt Dt Ais Pet SOR Pepin 9! Heep Hyoyr sy ‘. 4! Pe Ly Sy! wed ets tere’, we # et eee ye 
ae he ts ea Y ae et Eee $a eo 3 fe i 31 cwetyn igts: Se: es pea “pues won teas 16 we BS ae os, +t -a'® Woe path 
Pee fu) bt Satis "6 woh nA 9%! 8 92, A “fia aes Gs rt Jere aie be Nei sayrect Pease ot- HK Ree GE fs , Seq teee or." 
} *; fe are ae! i ake med ete me t ett pateaeay, we Per 906 Bet ardy ne “es AAO to ws. vue ori Py 
8 Greens Py ~ aye ule ie Bare Ags ; pide Ph * , . rie te wie fea r Fee n °° yey? wee? EF SL & a ks : tt “bel 
nag ‘win : wee ones Hee ? + ‘¢t eA: f ee pee oe ge rt é I a 6. vw AC ayeke? begs e fen ra rs *{ le . oP A aise 
t vs AT rye Misc en ed ah ste * tytn an dhe Wen ohare cee nel agh tor 2 yah ewan Anes Qe: 
a tat it iets ° ew; pi: = t A et Wa + Span 3 Phe ay Se oc oF te ws +s ese F Pi | neta ad s 
te as > ye it Maes pa rit e, ae sac ie Vay Meas gk wtetstet hy, at batt Re teh gt ctarde 0 a ‘5:7 ee 
a wth *wotiey in Sy, ees ih patito i i tiie ESRC UA Se tan Wart ee Se er 
2 Whyte Paes hie . weegi ls ah ONE Te abel bees 0 tt ame ty arias sa an 
cers’, pee :. eats nee! Pts i ae ee m Rt } ish ha tn ate ts *re ity ee Anote ANS sraseuarsars sin ° y Fe Lt: ° a vagete pees, §° 
ae? napee ju b5 81 tn bed a Le 9049 he ie y, oes ao 9 BE gd yhk ee yy a: a. oe ho avi s° Sah eens eM 
Thy every eee ican fe eee! ote ance mae 4 CY & Deen *0¢: gery ob ‘ "hi Vady Shy xi. ae . a at A | 38 re “2 i. 
ST "y , a ri . re s% . e u Bee ° 
‘ 4 it ze Ane var ret dy arse rey H aici % col = ah perce S A, ath ’ a nee e hy OL 
ole" 5 eerle s cf ' » ° ‘ 
1h oe Viens : ae Gog MA He, a Sth ot? “ UH wit! iinet wi . a ae tint H hare 
nog th a i. ett . “a SS br v perigee aH * ete peek StREee or Come” 2, 8 eat vee fi ‘ Diane 
ft Ns Aj eee 
: pn dae preielitae RORY te bai is Eh TEA mas ert rea ety 
was oe ty Sette ‘ytyis c my oF} ait gy RSLs vs ,} Ut efdat mosh a5" ee Refeta aa ' i . ee ee 8 
Siior a Pty ey sy Bigy urea yet alee Wate oN ve tee Lee trneds « te SH aot: 
ahs Bee: re) 1c tie as or Ly 3 yaheb Leh Ts ta ef age. & mks jy te “bts “ats! ‘ | 
ar 5 ae nit aa as adeseit ara ad Pee gos arFk oe Bsa sf etagrar isd ow % meee i é Sache 
ae Siew OAT aE late AMMEN wot pane, oe sgtactie srefte 
. y De tre” ba i ‘ ce 4 | nares ‘a e tes @ @ oe fees 8. . . . = 
eyes hy chee Ay poate sf gat psifen citi " ante eh q aah a ae s 7} He . Pd SSS) 85 te pint e Pied . . e* 
tee 4 N Ash st a ve oa pees Pu re ean PU ‘ ane Sa : H : 
PCR by presses RS Penile “oe ee wey v3 spent Hh a eae ; bi ate np ate tee A 
a BSS, ie oP i Diep rie ‘3 Way beret pee ny nan Ke POURS atau Wysh ye +3 CORSON, Fon Ps ’ 
tah eae 8 Hox oe ar my Wtiont vests ti aie yet oe Cy ai iat r : oe Bre at) if oe t %, c 
o% . vee \ i Tete ae fg xk = Py pie Be eying: Wei oe ut 8 oh ah iy ras cf abet ee tes; coke eteuserswanucnce apts? wae 
SR ' de 1, ME ws afte hie vias tw operate v rt) ee tetefyt KS ine Pepneden vse 4 “Us 12 by a Sata Ce a S earh iave oe, 
eu HS tt Saal SRR g TKR a 2 ate ae LE Re aerr Pifeor ih Vat ae sh les B B 
“i meine pees Set bate eho? sayy 
oe ate jena He | ato® Ges vf » aye 
if gh atats “ hha beh fs hind sinh aT 
wie i} asi Mastin sey ata Big ad 
ow wi a je it #} % i tie a iste 8 M4 wi Bet sae ~e th : ‘ i: be Srey tpn tthe os 8 vk “ab ig aWe : ” “ 
ihe AY “$i ser 2s soi btetatet wtye oh Budge) eh tats) gon? a id fasts os itee ew bSer ye hat we mbzng ofp at oy a Roe 2 Tid Pee ACY ty heal Be an eee paras , : si eseee cog” e 5 e ay “een” = 
"ef iS Path " pa! 4} », stats Y eles oe ie wp rk Bey Leh ‘ sweet: ras ou) Dae pares aeare Atos rk & - al foie etd Ae Car | . 
Risen rod erat id Faye stale ys > hs Sot gP at wit 4! ne Sak ts ” rg8's yuyye a 4 f « ae LENE he ped r,t he) : ? ae Seopa . ay . a . 8 Y 
¥ SENSeH ay i ', BOAT At 2 ak petal “Oh et Sih hel eB aL ti . gy Uys alan eit “ 2 ates 8 $y e . © 
As ‘ A if ae Me vie : figet, ! at Perit oe tt ay an A DG oF af * ai hee AA Ae hattereny ts noth vAN stad “Tore i a; Oh Pe ree ee t pe EL Pa ay care 
Bat us ony i ict : Tit dadyce Ait th a es St wr) ‘ BDI e on » ‘* eta Ww vt pee tt ey eted s st ‘ Sitare oe . 
oth soins ’ a ice sa Pe 4s #) Fp le 4 rn aoe a et rue tet Os nt wt of * ve r) 
4 aye Fowler ENS 198 Bho ay eS} s 36h 44: ss ¢ Hae ee OTT es » 
Meena Genay Aesth } ’ pat tarees PP tind ed 5 i ‘ 4 
Hf i feietaho! nr M : ar aid. a es 
ete anh she ws Asay H : Bafytee ors Om acy bie ve , 2. yh 8 
Sie ae i ae rest SEUNG ee eh et Sere De 
ye a less nate Rie dys arises EMME aety aN He ae eT FR CS a nee gy” 
Poarisegtices k ait Pesat the teabaee he . 
yiy Falatvtateles ' i ae wiciateat 2 +, Rey in a i? ' ‘ ea ae i He i) aesras “ meh. a é : { : ; > . 4 
fy heel reeds" : Ane Bete As eu Bat (Sofie at wrethes ain i Citas ae 1 t ‘ . L ‘ i : : ts area | 
Nid whe Arges Me wat ri! ‘ Set Baty ad dA ee Wassectay: “ai uty ati a) Ng one 4 q 2 Ae iy , oa a of 
ds ie wae pin x iit eat tate yaaa e ae in vies ths Nite ei 
eee 2° oN 3 aN SH at os ite vale rie Has. ROKLE? th ut Pete Nese ra 3! pte e 
re! i he : svar ry oak TREN 3 ; . 
eet nation Hi Ml, 1 tenet ey i Vite tat ' yf ! ts at ite tek i nae ri " ne «eyed BaF we. 3 . : : ; tot -. 
ity Lae aa uhtab if) ae bs ey ay ahi ree tate esely : hi ple #99 S, ta he 8s nie oh Mie ‘ aeons att 
BAW nea’ % I tehy yee i er phe Posi ate wie aE ae Silber Oe Wan : bee’ bed ye ee} Pe | 
f Ay oe He into Hi tS Rit. Bh , «0 te O ate tia dt FP dad- Se yh) W 4 4 . ‘ 
te aL veh BANE Ph Es Rites ofp erates Pe Aaa AAC SR, MINS ve Eaa"ehe -— a : _. 
hy « 7 O Ad é ait eae i * ® 
se, vay ae i me debe Net aac Cee Rs Me re se yn 
. rae : iste He! Re ar ' a Scie , t tee eee a ase % 
£4) erie yo seh pi? aut Pel cr on Oc on* MP been ! ae ees Be ee ? “4 . > ‘ . ete ese 08 2 oe . 
i "| seul? tA des Farad, at ucts S a pees 8 roaste se he oe ee Ae gee Sonar a ry & . " 
ot Pegs ae 6h bia ait ARSE at as $. Bip ete Po € vee oe he . > ” eo. 
Ly ty ftiis Meunis ? niv t "° nee “ ; he a a Stee) wi ee +5 oh: pel “hi, > , Les ‘a 5 " LN s, ion 5 eae .* soi tn 
Bh Pe ‘ ea aid yee se ate tek UNI ths -wa Se ae ee 
é ae 4 . e . ; 
a Nate seat bess ear tate Patty The, , eC eee eet 
i) He aa ‘ wi ys sug Vaya! "5 ACs: At t e . 4 : ie a 3 Vidgrae ur ? e . 
ira wie \p Ae vis A) oe to, P ys es race eve : 2 » 
¢! a hie “{ ’ E30 dege, © tle subte = . oc _ , t zi 
Mi rt Baty 4: ome erat, ant see rf eve 2 ' 0? ah ai Ne = H ’ as . ry ny s . 
PR Rt rat i tq t ty ead - san nie Dee se : 
Bre eS RE t Ra Tae Le CORRE A NY) Sart seen apieaeecy) (me, rae 
LS dot be tk 2h ee ad ae ene aaa dt - 7) eas “Mey A . 8 i. 2 . 





